CIT 594 Errata #3 for Bugs Grammar (and Code)
Spring 2008, David Matuszek

This is a minor change, important only if you are using the followedBy(Recognizer, String) method in the supplied RecognizerTest.java file.

The problem comes about because the RecognizerTest.java file is recycled from last year, and doesn't customize the StreamTokenizer that it uses in quite the same way. One consequence is that it ignores any newline ('\n') characters in its parameter list, and there are undoubtedly other problems as well.

Rather than just copy the customizations from the Recognizer class, my preference is to obey the DRY principle and create the tokenizer in just one place, the Recognizer class. This means (1) making the createTokenizer method public, (2) refactoring the createTokenizer method to extract the customization code, and (3) having the createTokenizer method return the new tokenizer method, rather that just setting an instance variable. Finally, (4) the Recognizer constructor has to save the created tokenizer.

Was:
/**
 * Constructs a Recognizer for the given string.
 * @param text The string to be recognized.
 */
public Recognizer(String text) {
    Reader reader = new StringReader(text);
    createTokenizer(reader);
}

/**
 * Creates a StreamTokenizer for the given <code>reader</code>
 * and sets a bunch of parameters for it.
 * 
 * @param reader The input source to be tokenized.
 */
private void createTokenizer(Reader reader) {
    tokenizer = new StreamTokenizer(reader);
    tokenizer.eolIsSignificant(true);
    tokenizer.slashStarComments(true);
    tokenizer.slashSlashComments(true);
    tokenizer.lowerCaseMode(false);
    tokenizer.ordinaryChars(33, 47);
    tokenizer.ordinaryChars(58, 64);
    tokenizer.ordinaryChars(91, 96);
    tokenizer.ordinaryChars(123, 126);
    tokenizer.quoteChar('\"');
}
    
Now:
    /**
     * Constructs a Recognizer for the given string.
     * @param text The string to be recognized.
     */
    public Recognizer(String text) {
        Reader reader = new StringReader(text);
        tokenizer = createTokenizer(reader);
    }

    /**
     * Creates a StreamTokenizer for the given <code>reader</code>
     * and sets a bunch of parameters for it.
     * 
     * @param reader The input source to be tokenized.
     * @return The customized tokenizer.
     */
    public StreamTokenizer createTokenizer(Reader reader) {
        StreamTokenizer tokenizer = new StreamTokenizer(reader);
        tokenizer.eolIsSignificant(true);
        tokenizer.slashStarComments(true);
        tokenizer.slashSlashComments(true);
        tokenizer.lowerCaseMode(false);
        tokenizer.ordinaryChars(33, 47);
        tokenizer.ordinaryChars(58, 64);
        tokenizer.ordinaryChars(91, 96);
        tokenizer.ordinaryChars(123, 126);
        tokenizer.quoteChar('\"');
        return tokenizer;
    }

To make use of this change, the following lines need to be changed in the followedBy(Recognizer, String) method:

Was:
StreamTokenizer expected = new StreamTokenizer(reader);
expected.ordinaryChar('-');
expected.ordinaryChar('/');
Now:
Reader reader = new StringReader(expectedTokens);
StreamTokenizer expected = recognizer.createTokenizer(reader);

Of course, I should test the change.

Now:
@Test
public void leeKuoTest() {
    Recognizer r = new Recognizer("color \n $");
    assertFalse(r.isDoStatement());
    followedBy(r, "color \n $");
}

(Thanks to Lee-Kuo Chen for pointing out this problem.)