CIT 594 Recognizer Improvements
Spring 2007, David Matuszek

Here are some things that may help you in writing the assignment. These are not new requirements.

I've improved my error(String message) method to give you more information, specifically, what tokens still remain (up to five of them) to be returned by the tokenizer.

    private void error(String message) {
        if (!nextTokenMatches(Token.Type.EOF)) {
            message += "\nRemaining tokens:";
        }
        for (int i = 0; i < 5; i++) {
            if (nextTokenMatches(Token.Type.EOF)) {
                pushBack();
                break;
            }
            else if (nextTokenMatches(Token.Type.EOL)) {
                message += "\\n";
            }
            else message += " " + nextToken();
        }
        if (!nextTokenMatches(Token.Type.EOF)) {
            message += " ...";
        }
        throw new SyntaxException(message);
    }

I did not expect this, but it turns out that there is at least one case where the word end causes trouble because it is mistaken for a variable. The best solution is to distinguish between "keywords" and other words. Here is code for this:

    String[] keywordArray = {
        "move", "turn", "take", "drop", "zap",
        "repeat", "while", "if", "else", "return",
        "north", "east", "south", "west", "left", "right", "around",
        "WALL", "ENEMY", "FUEL", "ARMOR", "ZAP_GUN",
        "seeing", "smelling", "facing",
        "and", "or", "not", "define", "end"
    };   
   List<String> KEYWORDS = Arrays.asList(keywordArray);

    public boolean isKeyword() {
        Token token = nextToken();
        if (KEYWORDS.contains(token.value)) return true;
        else {
            pushBack();
            return false;
        }
    }

To take advantage of this, modify isVariable() as follows:

    public boolean isVariable() {
if (isKeyword()) {
pushBack();
return false;
}
return name();
}

It might be convenient to add a keyword(expectedKeyword) method to go along with my number(), name(), name(expectedName), and symbol(expectedSymbol) methods. If you think so, it should be easy to write one.