Class BnfTokenizer

java.lang.Object
  BnfTokenizer

public class BnfTokenizer
extends java.lang.Object

A testbed for a tokenizer for BNF definitions. File StreamTokenizerAndTranslator.java Created on Jan 11, 2007

Constructor Summary

BnfTokenizer()
Constructs a BnfTokenizer with no text to tokenize; must call tokenize(String) before asking for tokens.

BnfTokenizer(java.lang.String text)
Constructs a BnfTokenizer with specific initial text to tokenize.

Method Summary

java.lang.String	nextToken() Tokenizes BNF definitions and returns each token as a String, one token per call.
void	tokenize(java.lang.String text) Sets the text to be turned into tokens by calls to nextToken().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BnfTokenizer

public BnfTokenizer()

Constructs a BnfTokenizer with no text to tokenize; must call tokenize(String) before asking for tokens.

BnfTokenizer

public BnfTokenizer(java.lang.String text)

Constructs a BnfTokenizer with specific initial text to tokenize.

Parameters:

text - The text to be tokenized.

Method Detail

tokenize

public void tokenize(java.lang.String text)

Sets the text to be turned into tokens by calls to nextToken().

Parameters:

text - The text to be tokenized.

nextToken

public java.lang.String nextToken()
                           throws java.lang.IllegalStateException

Tokenizes BNF definitions and returns each token as a String, one token per call.
Examples:

Input characters	Returned string
<nonterminal>	<nonterminal>
terminal	nonterminal
"terminal"	nonterminal
+	+
::=	::=
|	|
\n	\n
(end of input)	EOF

A single ':' is legal, but the sequence "::" must be followed by "=", or "[ERROR]" is returned.

Returns:

The next token.

Throws:

java.io.IOException - If unable to read the next token.

java.lang.IllegalStateException - If called without setting a string to be tokenized.