CIS 6300: Advanced NLP - Efficiency
Natural language processing - CIS 5300 or equivalent, Deep learning - CIS 5220 or equivalent, research experience highly recommend.
This course will focus on natural language processing with limited resources.
In current machine learning approaches, resources can be constrained along several dimensions, including hardware resources, or data resourses.
The course is meant as a survey of several sub-areas in efficient NLP, focusing first on data efficiency (few and zero shot methods), and model efficiency ( compression, quantization, and novel architectures).
We will primarily discuss recent advances in the form of published papers in the last 5-10 years.
The main goals are to understand existing literature, and find opportunities to apply methods to existing problem domains.
The course is research oriented, and students must be prepared to read and understand papers largely independently.
Every class period will either be
- Lecture / talk introducing a topic or background or recent work
- A discussion group in the form a roles seminar led by participants of the class
Tuesday / Thursday 10:15-11:45 EST in FAGN 118.
Attendance is mandatory, and sessions will not
(only by permission)
class participation (10%), discussion questions (10%), paper presenter role for reading group (20%), 2 x other presentation roles for a paper (30%), final project, done in pairs (30%)
Reading group sign up and paper list can be found on the here.
The roles seminar
is an approach desinged to help participants engage in different perspectives to analyze a paper.
The goal is to engage with the paper along different dimensions, to properly understand its impact.
In CIS 6300, groups of 3 will be assigned to lead a paper. One member will be the presenter, and then the other two must select from distinct the other roles described below.
For part of the time in the seminar, you must take the perspective of your role.
|Presenter|| Create the main presentation, describing the motivation, problem definition, method, and experimental findings of this paper. |
|Peer reviewer||Complete a full NEURIPS style review of the paper. It is important to be critical but not negative. Your goal is nuanced weighing of positive and negative arguments for the paper. |
|Futurist||Propose an imaginary follow-up project that builds on, or deepens the paper we are discussing. Focus on scientific extensions. |
|Social impact assesor|| Identify how this paper self-assessed its (likely positive) impact on the world. Have any additional positive social impacts been left out? What are possible negative social impacts that were overlooked or omitted? What has been the impact (good or bad) of this paper on the economy, society, and/or the environment? |
|Industry practitioner||Propose a new application for the method in the paper (not already discussed in class), and discuss at least one positive and negative impact of this application. |
|Archeologist||This paper was found buried under ground in the desert. You're an archeologist who must determine where this paper sits in the context of previous and subsequent work. Find and report on one older paper cited within the current paper that substantially influenced the current paper and one newer paper that cites this current paper. If the paper is too new to have signficant follow up, report on two previous works.|
The goal of the project is allow you to explore ideas you are learning about in more depth on a problem domain of your choice.
There will be three forms of a project:
|Survey || Complete a literature survey of efficiency in a problem formulation you are interested in. For example, you may survey work on zero-shot image captioning. Your presentation should organize the literature into groups, and present high level summaries of at least 3 important papers.|
|Reimplementation || Complete a partial reimplementation of a method in a domain you are interested in. You can use modern tools (i.e. pytorch), but you must implement all non-boiler plate code yourself. Your presentation should report on what you accomplished, and the main challanages for reproducing the result. |
|New Application || Propose a novel extension or application of the methods we are disucssing. Be sure to scope it appropriately for a semester. Your presentation should motivate the extension, and explain novel experiemnts.
- Project Proposal: A 1-2 page report establishing a title for your project, a motivation for the area you want to work in, and a summary of what you expect to accomplish in the project. Include a specific timeline, and what you expect to accomplish by the check-in.
Due 2/2, 5% of the grade.
- Project Update: A 2 page summary of progress so far, including changes to direction. This is your half-way point, so I expect you to have made signficant progress toward your project.
Due 3/16, 5% of the grade.
- Project Presentation: A 12 minute in-class presentation summarizing your project.
Due 4/20 and 4/25. 10% of the grade.
- Project Report: A 6 page neurips formated report.
Due 5/1. 10% of the grade