Machine Learning and Natural Language

Fall 2005

Course Information

Meeting Times and Locations:

Lecture:  Tuesday/Thursday 12:30 - 1:45, 1105 Siebel Center

Professor Dan Roth

Office: 3322 Siebel Center

Office Hours: Thursday 2-4 pm

Phone: (217) 244-7068

E-mail: danr at uiuc dot edu

Course Description

The study of the computational processes underlying comprehension and generation of natural language is an important scientific and engineering task. Instilling machines with abilities that allow them interact to intelligently with humans depends on increasing levels of natural language comprehension (and generation), although shallow levels of understanding can already successfully support challenging applications such as intelligent information extraction, automatic translation, summarization and others. At the same time, understanding how humans process language is a challenging and important scientific endeavor that may also shed some light on many language engineering tasks.
Realizing that natural language understanding tasks are knowledge intensive, in the sense that they depend heavily on ``real-world knowledge'', earlier lines of research in these directions relied heavily on manual knowledge preparation -- "world" knowledge as well as linguistics knowledge. It is generally accepted today that a learning component must have a central role in supporting natural language related tasks; learning processes have multiple roles, from knowledge acquisition to supporting non-brittleness when facing previously unseen situations. A significant amount of work has been devoted in the last few years to developing statistics based and learning methods for these tasks, with considerable success.

This course will introduce some of the central learning frameworks and techniques that have emerged in this field and found applications in several areas in text and speech processing: from information retrieval and extraction, through speech recognition to translation and text comprehension. We will present the main theoretical paradigms used in the field -- learning theoretic, probabilistic, and information theoretic -- the relations between them, and the main algorithmic techniques developed within these paradigms.
The course will be reading intensive as well as project intensive. We will introduce some of the fundamental ideas and discuss those by going through relevant papers and working on relevant projects.
This will be done in the context of thinking, and attempting to address, the problem that will serve as the final project -- story comprehension.

Tentative Course Plan

Prerequisites

Previous knowledge of NLP or AI is not required. Background in algorithms (CS225 or equivalent) and theory of computation(CS273 or equivalent) is required.

Course Materials

I will not follow a text book. Relevant papers and notes will be available from the course home page. The following texts contain some of the material that will be covered and some portions of them will be recommended reading.

    • Daniel Jurafsky and James H. Martin, Speech and Language Processing , Prentice Hall 2000
    • Christopher D. Manning and Hinrich Schutze, Foundations of Statistical Natural Language Processing , MIT Press 1999
    • Eugene Charniak, Statistical Language Learning, MIT Press 1993
    • Frederick Jelinek, Statistical Methods for Speech Recognition, MIT Press 1998
    • Steve Young and G. Bloothooft (Eds), Corpus-Based Methods in Language and Speech Processing, Kluwer
    • James Allen, Natural Language Understanding, Addison-Wesley

Assignments

    There will be (1) several programming and experimental assignments; (2) reading assignments (3) at least one presentation (ideally, more) and (4) a final project. There is no final exam.
    The current plan for experimental assignments is:
    • Statistics: A small individual project - TBA
    • Classification: A small individual project - TBA
    • Story Comprehension: A group project: A base line system
    • Story Comprehension: A group project A final Project

    Reading and Presentations: Mandatory readings and additional recommended readings will be assigned every week.
    • Four (4) times a semester you will write a short critical essay on one of the additional readings.
    • Once a semester you will present a paper (15 min presentation).

    Assignments and Project Presentations: Experimental assignments and the final project will be done in groups. Each group will present its work.
    More details will be given along with the assignments.

Expectations

This is an advanced course. I view my role as guiding you through the material and helping you in your first steps as an researcher. I expect that your participation in class, reading assignments and experimental assignments will reflect independence, mathematical rigor and critical thinking.

Dan Roth