CIS 630 -- Machine Learning for Language Processing

Prerequisites

Syllabus

• Survey applications of machine learning to language processing
• document classification
• document segmentation, tagging, and entity extraction
• parsing
• inducing representations of linguistic objects
• possible guest lectures (depending on guest availability):
• dimensionality reduction methods
• non-negative factorizations
• finite-state techniques
• the latest on probabilistic parsing
• reinforcement learning in dialog systems
• Introduce techniques and open research questions
• Generative vs discriminative:
• modeling the joint distribution of labels and documents
• naive Bayes
• Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval
• A Comparison of Event Models for Naive Bayes Text Classification
• belief nets
• Learning Limited Dependence Bayesian Classifiers
• Hierarchically Classifying Documents using Very Few Words
  
• modeling the conditional distribution of labels given documents
• conditional maximum entropy
• A Maximum Entropy Approach to Natural Language Processing
• A Maximum Entropy Model for Prepositional Phrase Attachment
• Using Maximum Entropy for Text Classification
• comparing generative and discriminative probabilistic models
• On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes
  
• minimizing (bounds on) classification error
• maximum margin (boosting, SVM)
• BoosTexter: A Boosting-Based System for Text Categorization
• Boosting and Rocchio Applied to Text Filtering
  
• Text Categorization with Support Vector Machines: Learning with Many Relevant Features
• online (winnow)
• Mistake-Driven Learning in Text Categorization
• Learning to Resolve Natural Language Ambiguities: a Unified Approach
• Limited training data
• smoothing and model averaging
  
• A Gaussian Prior for Smoothing Maximum Entropy Models
• An Efficient Extension to Mixture Techniques for Prediction and Decision Trees
  
• Learning Theory and Language Modeling
  
• how sensitive are different methods to small samples, noise?
• Learning in Natural Language  
• using unlabeled data
• Combining Labeled and Unlabeled Data With Co-Training
• Text Classification from Labeled and Unlabeled Documents using EM
• modeling sequences: probabilistic mappings vs local classifiers
• joint models: hidden Markov Models
• An Algorithm that Learns What's in a Name
• Information Extraction with HMMs and Shrinkage
• conditional models: maximum-entropy Markov models and random fields
• A Maximum Entropy Model for Part-of-Speech Tagging
• Maximum Entropy Markov Models for Information Extraction and Segmentation
• Conditional Random Fields
• partial labeling: expectation-maximization with the forward-backward algorithm
• local classifiers: boosting, winnow
• Boosting Applied to Tagging and PP Attachment
• A Winnow-Based Approach to Context-Sensitive Spelling Correction
  
• The Use of Classifiers in Sequential Inference
• creating representations by unsupervised learning
• latent variable models
• Aggregate and Mixed-Order Markov Models for Statistical Language Processing
• Probabilistic Latent Semantic Indexing
• Latent Dirichlet Allocation
• modeling linguistic relations
  • EM-Based Clustering for NLP Applications
    
  • Clustering Verbs Semantically According to their Alternation Behaviour
  • Using a Probabilistic Class-Based Lexicon for Lexical Ambiguity Resolution
    
• information bottleneck
• The Information Bottleneck Method
• smoothing
• Distributional similarity models: Clustering vs. nearest neighbors
• Document Clustering using Word Clusters via the Information Bottleneck Method
• feature induction
• The Power of Word Clustering for Text Classification
• On feature distributional clustering for text categorization
  

• Practice research methods
• surveying the literature
• summarizing mathematical models and algorithms
• identifying workable open questions
• designing informative experiments
• reporting results effectively
  
• Produce reusable results
• survey of the literature
• Example: http://lcg-www.uia.ac.be/~erikt/research/np-chunking.html
  
• "gentle introduction to X"
• Example:The Improved Iterative Scaling Algorithm: a Gentle Introduction
  
• practical implementation of previously impractical algorithm
• critical data set for evaluating an important hypothesis
• publishable paper (conference deadlines coming up early 2002!)

Format (tentative)

• What I do:
• Give introductory lectures on each topic
• Select appropriate papers
  
• Steer discussion
• Comment when necessary
• Suggest and advise on projects
• Organize a course Web site with links to course products
  
• What you do:
• Individually:
• Present a comparative discussion of a few papers on a given topic
• Take responsibility for a well-defined part of a project
• Serve as discussant of a project other than yours
• Individually or as member of a work group:
• Present a project proposal (15-20 minutes)
• Work on the project
• Present a mid-term project status report (10-15 minutes)
• Write a project report accessible on the Web to the class
  
• Present the project report (15-20 minutes), discussed by pre-appointed discussants
  

Grading (if you must know)