StringTokenizer in Java with Examples

In this guide, we will discuss StringTokenizer in Java.

StringTokenizer

The primary purpose of this class is provide methods to split string in java. The substrings generated after splitting are referred as tokens.

A Simple Example of StringTokenizer Class

Let’s see a simple example, where we are using StringTokenizer class to split a given string using the space as a delimiter.

import java.util.StringTokenizer;
public class JavaExample{
  public static void main(String args[]){
    String str = "Welcome to BeginnersBook.com";

    //delimiter is whitespace " "
    StringTokenizer strTokens = new StringTokenizer(str," ");

    System.out.println("Tokens after splitting: ");
    while (strTokens.hasMoreTokens()) {
      System.out.println(strTokens.nextToken());
    }
  }
}

Output:

Constructors of StringTokenizer class

As shown in the above example, we are using the constructor of this class to specify the input string and delimiter.
1. public StringTokenizer​(String str):
This constructor only require string parameter and uses the default delimiter set such as ” \t\n\r\f”: the space character, the tab character, the newline character, the carriage-return character etc. Delimiter characters themselves will not be treated as tokens.

2. StringTokenizer(String str, String delim):
Input string: str = “Welcome to BeginnersBook.com”
Constructor used: new StringTokenizer(str,” “)
Output: Three tokens “Welcome”, “to”, “BeginnersBook.com”

3. StringTokenizer(String str, String delimiter, boolean flag):
You can also pass a boolean flag (true or false) while calling the StringTokenizer constructor.

Example 1: If the flag is false
Input string: Given string is “hello guys” and delimiter is ” ” (space)
Constructor used: StringTokenizer strToken = new StringTokenizer(str, ” “, false);
Output: Tokens are “hello” and “guys”. //two tokens

Example 2: If the flag is true
Input string: Given string is “hello guys” and delimiter is ” ” (space)
Constructor used: StringTokenizer strToken = new StringTokenizer(str, ” “, true);
Output: Tokens are “”hello”, ” ” and “guys”. //three tokens

Example: StringTokenizer with Multiple Delimiters

Let’s see another example, where we are breaking the strings based on multiple delimiters.

import java.util.StringTokenizer;
public class JavaExample{
  public static void main(String args[]){
    //string with multiple delimiters slash, dot, comma
    String str = "hi/hello,bye.goodbye";

    //passing multiple delimiters
    StringTokenizer strTokens = new StringTokenizer(str, "/,.");
    while(strTokens.hasMoreElements()){
      System.out.println(strTokens.nextToken());
    }
  }
}

Output:

hi
hello
bye
goodbye

Methods of StringTokenizer class

• boolean hasMoreTokens(): Returns true if the next token is available.
• String nextToken(): It returns the next token.
• String nextToken(String delimiter): It returns the next token based on the delimiter.
• boolean hasMoreElements(): Similar to hasMoreTokens() method. Returns true if the token is available.
• Object nextElement(): Same as nextToken() except that it returns object type instead of string. It exists so that this class can implement the Enumeration interface.
• int countTokens(): It returns the count of number of tokens generated after string split.

Example of hasMoreTokens() and nextToken() Methods

We are using hasMoreTokens() method as the method condition, so that the loop continues to run until all the tokens are traversed. Inside loop, we are using nextToken() method to print the token on every loop iteration.

import java.util.StringTokenizer;
public class JavaExample{
  public static void main(String args[]){
    //comma separated string
    String str = "this,is,test,string";

    //delimiter is comma ","
    StringTokenizer strTokens = new StringTokenizer(str, ",");

    System.out.println("Displaying Tokens: ");
    while (strTokens.hasMoreTokens()) {
      System.out.println(strTokens.nextToken());
    }
  }
}

Output:

Example of countTokens() method

This method returns the count of number of tokens.

import java.util.StringTokenizer;
public class JavaExample{
  public static void main(String args[]){
    String str = "BeginnersBook.com";

    //delimiter is dot "."
    StringTokenizer strTokens = new StringTokenizer(str, ".");
    int count  = strTokens.countTokens();
    System.out.println("Number of tokens available: "+count);
  }
}

Output:

Number of tokens available: 2

Frequently Asked Questions:

Reference docs:

StringTokenizer