CIT 594 Assignment 4: Tokenizer
Spring 2009, David Matuszek

Purposes:

• To familiarize you with the façade design
• To create some classes that will be needed in later assignments

General Idea:

Write a program that, given a java.io.Reader, reads a stream from it and returns a series of tokens.

• A StringReader will allow you to read Strings; this is good for testing purposes.
• A FileReader will allow you to read from files; this is how your Tokenizer is most useful.

A token is one of: A word (a name or a keyword), a number, a punctuation mark, an end-of-line, or an end-of-input.

Write your program using Eclipse. Provide complete JUnit tests for your program, and document it fully with javadoc comments.

Details:

Write the following classes:

TokenType

This class is defined as:
enum TokenType { NAME, KEYWORD, NUMBER, SYMBOL, EOL, EOI, ERROR }

Token

A Token has a type and a value. The value is a String containing the exact characters that make up the Token. The type is a TokenType that tells what kind of thing the Token represents--a variable, keyword, number, or symbol (punctuation mark). End-of-lines and the (one) end of input are also returned as tokens. These have the following meanings:

• NAME -- begins with a letter or an underscore, consists of letters, digits, and underscores
• KEYWORD -- same as a NAME, but in addition is a value that occurs in a keywords array
  • Define this array in one place, and populate it with a handful of keywords form Java--if, else, while, do, for, switch, case, maybe a few others
• NUMBER -- one or more digits (only), possibly including a decimal point, but not including a sign.
• SYMBOL -- any single punctuation mark (not including whitespace)
• EOL -- an end of line character (value should be "\n")
• EOI -- the end of input (value can be an empty string, "")
• ERROR -- an error (for now, trying to get another token after getting an EOI)

The token class should also include the following constructors/methods:

• public Token(TokenType type, String value) -- constructor
• public String getValue() -- a getter method for the token's value
• public TokenType getType() -- a getter method for the token's type
• public boolean equals(Object o) -- a test for equality of tokens

Tokenizer

The Tokenizer will have at least the following constructors/methods:

• public Tokenizer(Reader reader) -- constructor (sets the String or File to be tokenized)
• public boolean hasNext() -- returns true if there are more tokens to be returned.
• public Token next() -- returns the next token from the string.
• public void backUp() -- "backs up" one token, so that whatever was returned from the most recent call to next() will be returned again the next time next() is called; only one token is remembered, so if you call backUp() multiple times, the second and subsequent calls don't make any difference

The Tokenizer should ignore (skip over) all types of comments.

TokenTest

A JUnit test class for Token. Pretty simple.

TokenizerTest

A JUnit test class for your Tokenizer. Be very careful, and be aware that some characters may need to be marked "ordinary."

Comments:

Write your Tokenizer class as a façade for the StreamTokenizer class. That is, your constructor for Tokenizer will create an instance of StreamTokenizer that will subsequently be used by the Tokenizer methods.

You don't need a main method. All your testing can be done via JUnit.

The Token and Tokenizer classes should be quite simple. You will probably find the TokenizerTest class much more challenging than the Tokenizer class itself. There are two significant advantages to having the JUnit tests, though:

• If you have a thorough set of tests (and if you have read and followed the assignment carefully), you will almost certainly get a perfect score on this assignment.
• Later assignments will almost certainly require modifications to the Token and Tokenizer classes, and the JUnit tests will be very helpful in making those modifications.

Due date:

Tuesday, February 16, before midnight (zipped and submitted via Blackboard).