The current assignment requires some minor changes in the Type and Tokenizer classes. My apologies; I can't forsee everything. Fortunately, you should have good JUnit tests, so these modifications should not be painful. They took me about 20 minutes, and that includes writing up this page.

Type

Add a new type END_OF_INPUT.

The type END_OF_INPUT will have the same meaning as the type EOL did in the original assignment. EOL will now mean "end of line", that is, a newline character, '\n'. That's how I should have defined it all along. Please update your Javadoc comments to reflect this change.

Token

It is unlikely that you will have to change Token class.

Tokenizer (and TokenizerTest)

Start by updating TokenizerTest. Basically, this requires two steps:

1. Replace all tests for an EOL token with tests for an END_OF_INPUT token.
2. Anyplace you have a newline character in an input string, add code to recognize it (rather than just skipping over it, as you did before).

Don't forget to run your tests and make sure they fail!

Now update your Tokenizer. For me, this meant adding a couple of lines to my state machine's initial state, changing my skipWhitespace() method to not skip over newlines, and changing next() to return END_OF_INPUT instead of EOL when it reached the end of the input.

Use JUnit to make sure your updated Tokenizer works properly.