0. Tutorial on Machine Learning Tools in NLP (8/24)

    I.    Introduction

Introduction to the Class , Notes (9/6)

Reading:

• Story Comprehension Information: Deep Read: A Reading Comprehension System
More project related papers:
• The Brown System
• The Utah System
• The Purdue System
• The Mitre System
• The Zurich argument


• Papers on Textual Entailment (Final Project likely topic): Proceeding of the "Recognizing Textual Entailment" Challenge 2005



Recommended Reading on Textual Entailment:

• Introductory Paper from the above Proceedings.
• Oren Glickman, Ido Dagan and Moshe Koppel. , A Probabilistic Classification Approach for Lexical Textual Entailment, AAAI (2005)
• R. Braz, R. Girju, V. Punyakanok, D. Roth, and M. Sammons , An Inference Model for Semantic Entailment and Question-Answering. AAAI (2005)
• Rajat Raina, Andrew Y. Ng, and Christopher Manning , Robust textual inference via learning and abductive reasoning. AAAI (2005).
• S. Abney, Statistical Methods and Linguistics (1996) In ``The Balancing Act, Judith Klavans and Philip Resnik, eds, MIT Press.

Additional Recommended Reading:

• Some tools: K. Church Unix for Poets
• Lillian Lee ``I'm sorry Dave, I'm afraid I can't do that'': Linguistics, Statistics, and Natural Language Processing circa 2001

Natural Language and Statistics (9/7)

Reading:

• Manning and Schutze, Chp. 1

Additional Recommended Reading:

• W. Nelson Francis and Henry Kucera, 1982. Frequency Analysis of English Usage. Houghton Mifflin.
• Of interest:The Brown Corpus Manual
• The Zipf Law Page

Introduction to Linguistics and its role in Natural Language Processing
Martha Palmer's slides (2004)(9/13, 9/15)

Reading:

• Manning and Schutze, Chp. 3: Linguistics Essentials

Additional Recommended Reading:

• James Allen, Chp. 1,2
• Charniak, Chp. 1
• G. Miller, WordNet: a lexical database for English.'' In: Communications of the ACM 38 (11), November 1995, pp. 39 - 41.
• WordNet, WordNet: Five papers on WordNet

II.   Mathematical Preliminaries

Mathematical Preliminaries and Computational Paradigms (9/15, 9/20)

Reading:

• Manning and Schutze, Chp. 2: Mathematical Foundations

Additional Recommended Reading:

• Charniak, Chp. 2
• K. Church A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text

Students Presentation:

• (9/20) David Killian: Light et. al Deep Read: A Reading Comprehension System
• (9/22) Dustin Parr: Oren Glickman, Ido Dagan and Moshe Koppel. , A Probabilistic Classification Approach for Lexical Textual Entailment, AAAI (2005)
• (9/27) Quang Do Xuan: K. Church A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text



III.    Statistics: Representation-less Approaches

Statistical Estimation: Ngrams and Backoff Models (9/23, 9/28)

Reading:

• Manning and Schutze, Chp. 5, 6
• K. Church Ngrams
• Slava M. Katz, (1987) Estimation of Probabilities from Sparse Data for the Language Model Component of a Speech Recognizer. IEEE Transactions on Acoustics, Speech and Signal Processing, ASSP-35 (3), pp 400--401

Additional Recommended Reading:

• Stanley F. Chen and Joshua Goodman (1996) . An empirical study of smoothing techniques for language modeling Proceedings of the 34th Meeting of the Association for Computational Linguistics, pp 310--318
• Gale and Sampson (Note) . Good-Turing Smoothing Without Tears
• Kenneth W. Church and William A. Gale, (1991) A comparison of the enhanced Good-Turing and deleted estimation methods for estimating probabilities of English bigrams. Computer Speech and Language 5, pp. 19--54.

Students Presentation:

• (9/29) Hahn Koo: Dekang Lin (1999) Automatic Identification of Non-compositional Phrases, ACL'99, pp.317-324.
• (10/4) Hui Fang: Hearst, M. (1992) Automatic Acquisition of Hyponyms from Large Text Corpora, Proceedings of the Fourteenth International Conference on Computational Linguistics, July 1992.



Statistical Similarity and Clustering (9/30)

Reading:

• Peter F. Brown and Vincent J. Della Pietra and Peter V. deSouza and Jennifer C. Lai and Robert L. Mercer, (1992) Class-based n-gram models of natural language. Computational Linguistics 18(4), pp. 467--479.
• Ido Dagan, Lillian Lee, and Fernando Pereira. (1999) Similarity-Based Models of Word Cooccurrence Probabilities. Machine Learning 34(1-3), special issue on natural language learning
• Julie Weeds, David Weir and Diana McCarthy (2004) Characterizing Measures of Lexical Distributional Similarity, Coling'04
• Dekang Lin (1998) An Information-Theoretic Definition of Similarity. ICML'98
• Dekang Lin and Patrick Pantel (2001) Discovery of Inference Rules for Question Answering. Natural Language Engineering 7(4):343-360.

Additional Recommended Reading:

• Hindle and Rooth, (1993) Structural Ambiguity and Lexical Relations. Computational Linguistics, 1993.
• Hinrich Schütze, (1992) Dimensions of Meaning. Proceedings of Supercomputing, pp 787-796.
• Dekang Lin and Patrick Pantel (2002) Discovering Word Senses from Text. KDD'02
• Gigju, Moldovan, Tatu, Antohe (2004) On the Semantics of Noun Compounds (Draft)
• Michael Collins and James Brooks, (1995) Prepositional Attachment through a Backed-off Model. Third Workshop on Very Large Corpora, pp. 27--38.
• Maayan Geffet and Ido Dagan (2004) Automatic Feature Vector Quality and Distributional Similarity, Coling'04.
• Christiane Fellbaum, ed., 1998. WordNet: An Electronic Lexical Database
• Annotated Bibliography for MultiWord Expressions (updated: 2002)
• The WordNet Page
• The Extended WordNet Page

Students Presentation:

• (10/6) Shailesh Kochhar: Peter F. Brown and Vincent J. Della Pietra and Peter V. deSouza and Jennifer C. Lai and Robert L. Mercer, (1992) Class-based n-gram models of natural language. Computational Linguistics 18(4), pp. 467--479.
• (10/18) Mike Yin: Dekang Lin and Patrick Pantel (2002) Discovering Word Senses from Text. KDD'02
• (10/??) Adam Vogel: Patrick Pantel and Deepak Ravichandran (2004) Automatically Labeling Semantic Classes. NAACL'04
• (10/20) Alex Sorokin: Toutanova, Manning, Ng, (2004) Learning Random Walk Models for Inducing Word Dependency Distributions, ICML'04

IV.    Learning Classifiers

Introduction to Classification(10/5, 10/12)

Reading:

• D. Roth, Learning to Resolve Natural Language Ambiguities: A Unified Approach
• L. Marquez, Machine Learning and Natural Language Processing

Additional Recommended Reading:

• C. Cardie and R. Mooney, Guest Editors' Introduction: Machine Learning and Natural Language Processing. Machine Learning Journal. Special Issue on Natural Language Learning. Machine Learning Journal 34(1/2/3), 1999
• A. Golding and D. Roth, A Winnow-Based Approach to Context Sensitive Spelling Correction In: Machine Learning Journal 34(1/2/3), 1999. Special Issue on Natural Language Learning.
• C. Cardie Embedded Machine Learning Systems for Natural Language Processing. A general Framework. IJCAI'95
• R. J. Mooney Inductive Logic Programming for Natural Language Processing ILP'97
• L. R. Tang and R. J. Mooney Automated Construction of Database Interfaces: Integrating Statistical and Relational Learning for Semantic Parsing. EMNLP'00
• D. YarowskyDecision Lists for Lexical Ambiguity: Application to Accent Restoration in Spanish and FrenchACL'94
• R. E. Schapire and Y. Singer BoosTexter: A boosting-based system for text categorization. Machine Learning, 39(2/3):135-168, 2000.
• T. Joanchims Text Categorization with Support Vector Machines: Learning with Many Relevant Features. ECML'98
• Veenstra, van den Bosch, Buchholz, Daelemans, Zavrel Memory-Based Word Sense Disambiguation. Computers and the Humanities, 2000
• Y. Even-Zohar and D. Roth, A Sequential Model for Multi Class Classification
• X. Li and D. Roth, X. Lin and D. Roth, ``Learning Questions Classifiers: The Role of Semantic Information. Natural Langauge Engineering, to appear.



Probabilistic Classifiers: NB, Max Entropy (10/18, 10/20, 10/27, 11/1)

Reading:

• Adam's Berger Maximum Entropy Page
• A. L. Berger, S. A. Della Pietra, V. J. Della Pietra, A Maximum Entropy Approach to Natural Language Processing, Computational Linguistics, 1996, 22(1), pp 39-71.

Additional Recommended Reading:

• A. Garg and D. Roth Understanding Probabilistic Classifiers. ECML'01
• Adwait Ratnaparkhi: A Maximum Entropy Model for Part of Speech Tagging
• Rob Malouf: A comparison of algorithms for maximum entropy parameter estimation
• Tom Minka's page on logistic regression

Students Presentation:

• (11/1) Liam Moran: E. Brill, Transformation based error driven learning and Natural Language Processing: a case study in part of speech tagging. Computational Linguistics 21(4), 1995
• (11/3) Bin Tan: Buchholz, Veenstra, van den Bosch Cascaded Grammatical Relation Assignment. EMNLP'99
• (10/8) Jing Jiang: Ratnaparkhi, Reynar, Roukos: A Maximum Entropy Model for Prepositional Phrase Attachment
• (11/10) Allen Chang: A. Ng and M. Jordan, On Discriminative vs. Generative Classifiers. A comparison of Logistics Regression and naive Bayes.



Discriminatory Approaches:


• Linear Threshold Units (Nick Rizzolo, 10/25)
• Classification; Features and Kernels (11/3) (Additional Notes: On-Line)
• SVM ;

Reading:

• D. Roth, (1998) Learning to Resolve Natural Language Ambiguities: A Unified Approach
• D. Roth Learning in Natural Language IJCAI'99
• A. Ng and M. Jordan,On Discriminative vs. Generative Classifiers. A comparison of Logistics Regression and naive Bayes.

Additional Recommended Reading:

• T. Zhang (2000) Statistical Behavior and Consistency of Methods based on Convex Risk Minimization
• T. Zhang Regularized Winnow Methods
• T. Zhang Large Margin Winnow Methods for Text Categorization
• Collins, Schapire, Singer (2000) Logistic Regression, AdaBoost and Bregman Distances
• Schapire, Singer (2000) BoosTexter: A Boosting-based System for Text Categorization
• F. Damerau, T. Zhang, S. Weiss, N. Indurkhya Text Categorization for a Comprehensive Time Dependent Benchmark

Students Presentation:

• (11/15) Eric Bengston: A. Golding and D. Roth, A Winnow-Based Approach to Context Sensitive Spelling Correction In: Machine Learning Journal 34(1/2/3), 1999. Special Issue on Natural Language Learning.




Kernels (Additional Notes: Tree Kernels (by Mike Collins)) (11/10)

Reading:

• Collins, Duffy (2002) Convolution Kernels for Natural Language
• Cumby, Roth (ICML'2003) On Kernel Methods for Relational Learning

Additional Recommended Reading:

• Khardon, Roth, Servedio (NIPS'2001) Efficiency versus Convergence of Boolean Kernels for On-Line Learning Algorithms

Students Presentation:

• (11/17) Presentation of PPA Assignment
• (11/29) Deng Cai: Taku Kudo and Yuji Matsumoto(ACL 2003): Fast methods for kernel-based text analysis



Semi-Supervised Learning; EM (Additional Notes: EM for HMMs (Notes by Mike Collins)) (Not Given)

Reading:

• A. Blum, T. Mitchell (COLT'99) Combining Labeled and Unlabeled Data with Co-Training

Additional Recommended Reading:

• M. Collins and Y. Singer Unsupervised Models for Named Entity Classification
• A. Grove and D. Roth, Machine Learning, 2001 Linear concepts and hidden variables.

Students Presentation:

• (12/01) Chih-Wei Hsu: D. Yarowsky (ACL'95) Unsupervised Word Sense Disambiguation Rivaling Supervised Methods

V.    Inference

From Multi-Class Classification to Structured Output Learning (11/16)

Reading:

• V. Punyakanok and D. Roth Inference with Classifiers: The Phrase Identification Problem.
• Rabiner, L. R., A tutorial on hidden Markov models and selected applications in speech recognition, Proceedings of the IEEE, 1989, vol. 77, no. 2, pp 257--285.
• Bengio, Markovian Models for Sequential Data
• D. Roth and W. Yih A Linear Programming Formulation for Global Inference in Natural Language Tasks. CoNLL'04

Additional Recommended Reading:

Students Presentation:




Inference with Classifiers; Structured Output; Semantic Parsing(12/2 -- 12/9)

Reading:

• J. Lafferty, A. McCallum, F.Pereira , Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence DataICML'02
• D. Roth, W. Yih , Integer Linear Programming Inference for Conditional Random Fields ICML'05.
• V. Punyakanok, D. Roth, W. Yih, and D. Zimak , A Linear Programming Formulation for Global Inference in Natural Language Tasks. COLING'04.

Additional Recommended Reading:

• M. Collins , Discriminative Training for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms EMNLP'02.
• D. Roth and W. Yih, A Linear Programming Formulation for Global Inference in Natural Language Tasks CoNLL'04.
• M. Collins , Parameter Estimation for Statistical Parsing Models: Theory and Practice of Distribution Free Methods Book Chapter.
• X. Carreras, L. M`arquez, J. Castro , Filtering-Ranking Perceptron Learning for Partial ParsingMachine Learning; to appear

Students Presentation:

• (12/6) Alexander Kotov: J. Lafferty, A. McCallum, F.Pereira , Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence DataICML'02
• (12/6) Ming-Wei Chang: M. Collins , Discriminative Training for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms EMNLP'02.
• (12/8) Arkady Epshteyn: M. Collins Three Generative Lexicalized Models for Statistical Parsing

Advanced Topics:

• Introduction To Parsing (12/??)
• Introduction To Machine Translation
(11/??)

Reading:

• E. Charniak Statistical Techniques for Natural Language Parsing

Additional Recommended Reading:

• E. Charniak Statistical Parsing with a Context-free Grammar and Word Statistics
• M. Collins Three Generative Lexicalized Models for Statistical Parsing
• M. Collins Discriminative Reranking for Natural Language Parsing
• S. Abney Stochastic Attribute-Value Grammars



VI.    Final Projects Presentations

Final Projects Presentation will take place on the day of the final exam; Monday (12/12); 4-7 pm 3405 SC

---

---

---