LDSCI7226 Natural Language Processing with Deep Learning Course Descriptor

Course Summary

Deep Learning has evolved as the leading artificial intelligence paradigm over the past decade providing us with the ability to learn complex functions from raw data at unprecedented accuracy and scale. Deep Learning has been applied to problems in object recognition, speech recognition, speech synthesis, forecasting, scientific computing, control, and many more.

This course has a particular focus on modern Deep Learning algorithms and sustainable development, it covers fundamentals of Natural Language Processing (NLP), word embedding models, recommender systems, expressivity, optimisation and generalisation, unsupervised deep learning and generative models, etc.

The course will be taught in the Python programming language and, in particular, the PyTorch machine learning framework.

Course Aims

The aims of this course are to:

• Systematically understand the fundamental concepts of Natural Language Processing (NLP) and Deep Learning.
• Consistently produce working programs that solve data analysis tasks, reason about them and/or improve their performance using techniques learned in the course.

Learning Outcomes

On successful completion of the course, students will be able to:

Knowledge and Understanding

Subject Specific Skills

Transferable and Professional Skills

Teaching and Learning

This course has a dedicated Virtual Learning Environment (VLE) page with a syllabus and range of additional resources (e.g. readings, question prompts, tasks, assignment briefs, discussion boards) to orientate and engage you in your studies.

The scheduled teaching and learning activities for this course are:

Lectures/Labs: Typically one lecture and one lab session per week:

• Version 1: All sessions in the same sized group, or
• Version 2: most of the sessions in larger groups; some of the sessions in smaller groups

Faculty hold regular ‘office hours’, which are opportunities for students to drop in or sign up to explore ideas, raise questions, or seek targeted guidance or feedback, individually or in small groups.

Students are to attend and participate in all the scheduled teaching and learning activities for this course and to manage their directed learning and independent study.

Indicative total learning hours for this course: 150

Employability Skills

• Communication skills
• Mathematical skills
• Programming skills

Assessment

Formative

Students will be formatively assessed during the course by means of set assignments. These do not count towards the end of year results but will provide students with developmental feedback. Set assignments will also amplify problem-solving skills and develop software components that form part of the coding assignments.

Summative

The assessment will consist of two written coding assignments, which the student will have to do according to the set guidelines for coding.

The coding assignments will be assessed in accordance with the assessment aims set out in the Programme Specification.

Feedback

Students will receive formal feedback in a variety of ways: written (including via email correspondence); oral (on an ad hoc basis) and indirectly through discussion during group tutorials. Students will also attend the formal meeting, Collections, in which they will receive constructive and developmental feedback on their  performance. 

Feedback is provided on summative written assignments which will be handed back to the students.

Indicative Reading

Note: Comprehensive and current reading lists for courses are produced annually in the Course Syllabus or other documentation provided to students; the indicative reading list provided below is used as part of the approval/modification process only.

Books

• Steven Bird, Ewan Klein, and Edward Loper. 2020. Natural Language Processing with Python. Online at www.nltk.org/book/
• Deep Learning for Natural Language Processing: Applications of Deep Neural Networks to Machine Learning Tasks by Pearson Learn IT
• Deep Learning with Python by Francois Chollet
• Aston Zhang, Zachary C. Lipton, Mu Li and Alexander J. Smola. 2020. Dive into Deep Learning. Online at https://d2l.ai

Journals

• Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781v3 [cs.CL]
• Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805v2 [cs.CL]
• Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. Attention Is All You Need. arXiv:1706.03762v5 [cs.CL]

Electronic Resources

• The Unreasonable Effectiveness of Recurrent Neural Networks, by Andrej Karpathy. Last accessed: October 2020
• Attention and Augmented Recurrent Neural Networks, by Chris Olah and Shan Carter. Last accessed: October 2020
• The Illustrated Transformer, by Jay Alammar. Last accessed: October 2020

Indicative Topics

Students will study the following topics: 

• Introduction to Natural Language Processing
• Word embeddings (e.g. the Skip-Gram, work2vec, ELMo and BERT models)
• Pretraining word embeddings
• Recommender systems
• Recurrent Neural Networks
• Expressivity, Optimization and Generalization
• Imbalanced classification and metric learning
• Unsupervised deep learning and generative models