Machine Learning and Natural Language
|Experimental Assignment I
Lexical Paraphrasing: A Statistical Study
- The assignment will be done in teams. There are 6 participating teams, and a managing team - the Preparation and Evaluation Team - that runs the experiment.
Your final report on the assignment is due on Friday,
Oct. 21. At this time, each team will turn in its results along with a
short report describing what you did, what were the
difficulties and what are your conclusions.
The Preparation and Evaluation will take as input the results given by other teams and analyze and compare them and report on Oct. 27.
Possibly, they will also give a shorter presentation earlier, to describe the data and provide details on how the evaluation will take place.
The final report, including a summary of the results of all teams will be due on Oct. 27. and
will also include a short presentation in class.
Feel free to send me e-mail or come to ask questions.
The goal of this problem set is to write a program that will identify verbs pairs with the following property: one of the two verbs can replace the other within some sentences such that the meaning of the resulting sentence will entail the meaning of the original one.
The assignment is based on the paper
- investigate two general approaches to this problem.
- propose an extension of your own.
O. Glickman and I. Dagan
Acquiring lexical paraphrases from a single corpus,,
In Recent Advances in Natural Language Processing III, Nicolov, Nicolas, Kalina Bontcheva, Galia Angelova and Ruslan Mitkov (eds.)
The assignment has 5 parts:
The Preparation and Evaluation Team will be responsible for the first and last stage. However, I will provide you with already preprocessed text.
The data is taken from the AQUAINT corpus, and it is already processed using Dekang Lin's dependency parser, minipar.
- Data Preparation and task definition.
- Developing and Evaluating the Glickman&Dagan algorithm.
- Developing and Evaluating the modified Lin algorithm.
- Proposing and Evaluating an extension.
- Final evaluation and a comparison of the teams results.
A preliminary snapshot of the data is available at here.
Some details on the files is at here.
This is a very large corpus; you may decide not to use all of it. Please note that the data is not freely available and id given to you only for the purpose of this project. Please do not place it anywhere that is freely accessible.
The Preparation and Evaluation Team will need to decide what other pre-processing need to be done, and they will will present these details in class on Oct. 11. They will also make the details
available from a web page on the course web page. At this stage, the task definition as well as the form of the output need to be clear to all other teams.
At the end, on Oct. 26, they will also submit a final document (and give a presentation)
which summarizes the results of all teams, compares the results of all teams on the common tasks, and
highlights other interesting experiments teams chose to perform.
In between, all other teams will do the the other stages. These reports will be due on Oct. 21.Do not wait until the
detailed presentation of data and task on Oct. 11 to start your work.
After you read this document and the relevant papers you have enough information to start your work.
Please note that the goal of stages (2) and (3) is to identify verbs pairs with the following property: one of the two verbs can replace the other within some sentences such that the meaning of the resulting sentence will entail the meaning of the original one.
That is, the task is not context sensitive. In each of these stages you will generate a list of verb pairs, and provide a way to evaluate how well you do.
For stage (4) you may design you own extension -- change the definition of the task in some way (e.g., make it context sensitive), extend the coverage of the algorithm, propose a different algorithm, etc.
Consequently, the output might be different, and you will have to choose it, and figure out a way to evaluate it.
Describe what you did, the specifics of your models, additional decision you had to make that are not detailed in the paper and the rational behind your decisions
Do the same for your proposed extension.
Report on the comparison between the two approaches.
Provide the code for your preprocessing and for your model estimation and evaluation.
Present the output of your program on the training corpus and the test corpus.
Package the code so that one can run it(details: P&E team)
We are interested in understanding the task (well defined? things have to be modified for one reason or another?) and a comparative study of the two approaches. When designing the experiments (P&E team) care should be taken that the comparison is fair.
If you decide to do other experiments, make sure you have something to compare with.
What to submit (Updated)
The P&E team will add here a detailed list of what is expected, and how to submit it.
Here are the details courtesy of the P&E Team.
Also, look here for Additional files
Your grade depends on:
The quality of your report
The quality of your results.
- Your originality in going beyond the minimal requirements.
Thursday, Oct. 21. (with some other due dates relevant to P&E).