Machine Learning and Natural Language

Fall 2005

Experimental Assignment II Classification: Prepositional Phrase Attachment

(Due 11/15/05)

General

The assignment will be done in teams. There are 6 participating teams (no Preparation and Evaluation Team this time)
Your final report on the assignment is due on Tuesday, Nov. 15. At this time, each team will turn in its results along with a short report describing what you did, what were the difficulties and what are your conclusions.
There may be a meeting devoted to the presentation of the result on 11/17.
Feel free to send me e-mail or come to ask questions.

Prepositional Phrase Attachment

Prepositional phrase attachment is a common cause of structural ambiguity in natural language. The task is to decide whether the Prepositional Phrase (PP) attaches to the Noun Phrase (NP), as in Buy the car with the steering wheel} or to the VP buy}, as in Buy the car with his money.

Earlier works on this problem (see the class lecture page for pointers to several relevant papers) considered as input the four head words involved in the attachment - the VP head, the first NP head, the preposition and the second NP head (i.e., buy, car, with and steering wheel}, respectively).

These 4-tuples, along with the attachment decision, constitute the labeled input sentence and will be the main input that you will be given in this problem set. The goal of this problem set is for you to train a classifier that is doing a good a job as possible resolving this ambiguity. In the course of doing it you will have a chance to

Generate several sets of features given that data (and possibly some external resources)
Experiment with several linear classifiers, and explore tuning the parameters of each of them.
Experiment with at least two possible settings of the problem
Write a short report describing in a clear and concise way (1) what you did and why; (2) what are your results, and (3) what are your conclusions.

For reference, the most recent paper I know of on this problem is: PP-attachment disambiguation using large context; , by Marian Olteanu and Dan Moldovan, in EMNLP/HLT 2005
The paper has a lot of pointers to relevant papers.

The Assignment

You will be given training and test data in the format mentioned above (and used in most papers on PPA) - the 4-tuple format.
Your assignment is to train classifiers on the training data, test it on the test data, and report your results.
You can use this data in order to generate features that will be given as input to your learning algorithm. For example, you can use as features all sub-sequences of the 4-tuple, total of 15 for every input sentence. You can use additional resources to generate better features. You can also use the whole sentence (I will supply the whole sentence along with the 4-tuples in case you want to extract feature from it).

As a minimum you need to do the following:

Use the 4-tuples data provided to generate at least 3 sets of features.

It is easy to generate features from the 4-tuple using FEX (look at the software packages available from the Cognitive Computation Group, but you can also generate them yourself).

Experiment with all three update rules available in the SNoW learning architecture: NB, Perceptron and Winnow.

In the case of Winnow and Perceptron, you need to experiment also with the Voted (averaged) perceptron/Winnow option and with the Regularization (thick separator) option and try to find optimal parameters.
In addition, you need to experiment with other parameters (promotion, demotion and initial weight are some of the options). The easiest way to do it is to use the tuning script available with the SNoW package.
For the case of NB, you can experiment with the smoothing parameter

You need to try to learn from all the training data, but also to experiment with the case in which you split the data and train each part separately. A natural split to consider is per preposition, but you can choose a different split (and justify it).

In addition, you can choose to compare with an additional classifier and/or to you more data (complete sentences rather than 4-tuples) and external resources for better features, but you do not have to do any of these.

Report

Describe what you did, the specifics of your experiments, models, additional decisions you had to make and justify the decisions you had made and your experimental design.
You will be running many experiments. Think about it ahead of time and be smart about how you do it, how do you collect your results from all the experiments and how do you present it. Both to yourself, so that you can figure out what is going on, and to me (or any other external reader).
Chose at least 3 interesting examples, good, bad, or just interesting for some reason, and discuss them in your write-up.
Conclude with some suggestions for improvements, future work, etc.

Data

The data is available here. It is already split to training and test data. Note that one mapping to features is also provided (in the SNoW input format), as an example. This is the mapping that makes use of all 15 conjunctions for each example; but, you will need to generate other mappings.

Grading

Your grade depends on:

The quality of your report
The quality of your results.
Your originality in going beyond the minimal requirements.

Due date

Tuesday, Nov. 15.

Dan Roth