Class BnfTokenizer

java.lang.Object
  extended by BnfTokenizer

public class BnfTokenizer
extends java.lang.Object

A testbed for a tokenizer for BNF definitions. File StreamTokenizerAndTranslator.java Created on Jan 11, 2007


Constructor Summary
BnfTokenizer()
          Constructs a BnfTokenizer with no text to tokenize; must call tokenize(String) before asking for tokens.
BnfTokenizer(java.lang.String text)
          Constructs a BnfTokenizer with specific initial text to tokenize.
 
Method Summary
 java.lang.String nextToken()
          Tokenizes BNF definitions and returns each token as a String, one token per call.
 void tokenize(java.lang.String text)
          Sets the text to be turned into tokens by calls to nextToken().
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

BnfTokenizer

public BnfTokenizer()
Constructs a BnfTokenizer with no text to tokenize; must call tokenize(String) before asking for tokens.


BnfTokenizer

public BnfTokenizer(java.lang.String text)
Constructs a BnfTokenizer with specific initial text to tokenize.

Parameters:
text - The text to be tokenized.
Method Detail

tokenize

public void tokenize(java.lang.String text)
Sets the text to be turned into tokens by calls to nextToken().

Parameters:
text - The text to be tokenized.

nextToken

public java.lang.String nextToken()
                           throws java.lang.IllegalStateException
Tokenizes BNF definitions and returns each token as a String, one token per call.
Examples:
Input characters Returned string
<nonterminal><nonterminal>
terminalnonterminal
"terminal"nonterminal
++
::=::=
||
\n\n
(end of input)EOF
A single ':' is legal, but the sequence "::" must be followed by "=", or "[ERROR]" is returned.

Returns:
The next token.
Throws:
java.io.IOException - If unable to read the next token.
java.lang.IllegalStateException - If called without setting a string to be tokenized.