uva-advanced-nlp

Final Project

In the final project, we would like to study the reproducibility of the deep learning methods proposed in recent publications. The goal of this project is to investigate the challenge and tricks to reproduce results in the papers.

Reproducibility is always an important issue in AI and any of its sub-fields, like machine learning and natural language processing. As the fields advance rapidly, reproducibility is the guarantee that we are standing on a solid ground. With the recent development of deep learning for NLP, this issue becomes even more serious, as we all understand that the performance of a deep learning model is sensitive to many factors, such as initialization methods, learning rates, and optimization methods.

Releasing research source code is one big step towards the direction of reproducibility. In this project, we are going to use some publicly available codes and try to reproduce the results in the related publications.

The goal of this final project is to investigate

• how easy or difficult it can be to reproduce the results with the released code?
• furthermore, how robust or brittle a model can be with respect to some factors like initialization, optimization methods, minibatch sizes etc.

Be honest about what you will find in the project!

1. Requirements of Choosing a Project

A selected project must meet the following requirements:

• it is a work accepted to a top NLP or machine learning conference (e.g., ACL, EMNLP, NAACL, ICML, NeurIPS/NIPS, and ICLR) in the past three years;
• it has released the source code publicly available on GitHub or Bitbucket;
• the release code has at least a brief instruction about how to use it;
• make sure you can download the data and find the specific data split either from the paper or the instruction.

2. Proposal Presentation

Proposal presentation will happen on Feb 2. Each student will have 15 minutes = 10 minutes for presentation and 5 minutes for QA.

A presentation should include two parts

• the first part briefly explain the basic idea of a selected project;
• the second part should explain why the selected project is interesting with respect to the reproducibility issue and how the student plans to conduct the work.

3. Midterm Presentation

The midterm presentation will be on Feb 8. Similar to the proposal presentation, each student will have about 10 minutes for presentation and 5 minutes for QA.

Each presentation should include the following two parts

• a brief review of the proposed project (about 2 minutes)
• a brief review of the proposed plan to reproduce the results (about 3 minutes)
• the progress that has been made so far (about 5 minutes)