|CIT 594 Assignment 1: Tokenizer
Spring 2006, David Matuszek
- To ensure you are familiar with Eclipse
- To ensure you are familiar with JUnit testing
- To ensure you are familiar with
- To familiarize you with
- To familiarize you with state-space machines
- To familiarize you with stacks
Write an API that, given a
String, breaks it into a series
of tokens. For our purposes, a token is one of: A word, a number, a punctuation mark, or an
Write your program using Eclipse, and document it fully with javadoc comments.
Write the following classes:
public enum Type
This enumeration should have the following values:
This is a very small class.)
Token has public fields
String value. The value holds
the exact characters that make up the
Token. The type
tells what kind of thing the
- A name begins with a letter, and may contain any number of letters,
digits, and/or underscores.
- A number consists of one or more digits. A sign (+ or -), if present,
is not part of a number.
- Any character that isn't whitespace and isn't part of a name or
- Denotes the "end of line." For this type, the
should be the empty string.
Note: To correctly recognize letters, digits, and
whitespace, use the methods in
The token class should also include the following constructor and methods:
public Token(Type type, String value) -- constructor
@Override public boolean equals(Object o) -- tests
if two Tokens are equal
@Override public String toString() -- Returns a String
representation of this Token.
@Override tells the Java 5 compiler
that you are trying to override an inherited method, so that it can
warn you if you get the signature wrong.
public Tokenizer implements Iterator
The Tokenizer will have at least the following constructor and methods:
public Tokenizer(String input) -- constructor (sets
the string to be tokenized).
public boolean hasNext() -- returns true if
there are more tokens to be returned.
public Token next() -- returns the next token
from the string.
public void remove() -- throws an
public void putBack(int howMany) -- steps back past
howMany tokens (so they can be returned
again by subsequent calls to
You don't need a
main method--all your testing can be done via
JUnit. Here are my JUnit tests, and you may
add more if you like; if you add any public methods to those required above,
you must provide JUnit tests for them.
- Implement your
Tokenizer.next() method with a state machine.
- Define your states as an inner
enum class within your
- Do not use
StringTokenizer or regular expressions
anywhere in this assignment.
- As you find tokens, put them on a
java.util.Stack, and when
you need to
putBack tokens, get them from this stack.
- Write any additional code needed to pass all the JUnit tests that I gave
you with this assignment.
Tuesday, January 17, before midnight (zipped and submitted via Blackboard).