Introduction to Deep Learning

Spring Semester 2024

Simon Fraser University

Instructor: Oliver Schulte

Breadth Area III

Contact Information Oliver Schulte

Office Location: TASC 1 9021.

Office Phone: 778-782-3390.

Office Hours:

• Public Office Hour: Tuesday 1-1:45pm
• Private Office Hour (with waiting room): Tuesday 1:45pm - 2:15pm Zoom Link

Email: myfirstname_mylastname@sfu[dot]ca

Contact Information Erfaneh Mahmoudzadeh (Ass. 1)

E-mail: ema61[at]sfu[dot]ca

TA Office Hour AND Location:

• Regular office hour: every Thursday, 12 pm - 1 pm at Zoom Link
• TA will have an extra office hour before each assignment deadline
  • Feb 5, 2024: Monday, 12 pm - 1 pm at Zoom Link
  • March 18, 2024: Monday, 12 pm - 1 pm at Zoom Link

Contact Information Abdolreza Mirzaei (Ass. 2)

E-mail: mirzaei[at]sfu[dot]ca

TA Office Hour AND Location:

• Regular office hour: every Monday, 12 pm - 1 pm at Zoom Link
• TA will have an extra office hour before each assignment deadline
  • Feb 16, 2024: Friday, 2 pm - 3 pm at [Zoom Link (https://sfu.zoom.us/j/81909349453?pwd=JI5VZHcPSLSZan1Swutr9D8rkXggw1.1)

Inactive links are under construction

Announcements

Class Format and Synchronicity

• My goal is to make the class as interactive as possible for a remote class.
  • Lectures will be synchronous, and interactive.
  • The exam will have a synchronous component.
  • On-line quizzes will be in synch with classes and open for an extended window (e.g. 10 hours).

Wait List

• Please check the department's FAQ before emailing me.
• I do not have control over the wait list
• I am unlikely to agree with you joining the course after the first week.

Course Resources

• Schedule. Updated March 11, 2024.
• Syllabus. Updated February 27, 2024.
• Textbook website.
• Installing Canvas Mobile. You need to install canvas to do the in-class quizzes.

• On-line Quizzes can also be found from within the course canvas site. I will announce quizzes in class. You are responsible for monitoring the site to be aware of upcoming ones.
• SFU Medical Excuse Form.

Slides

• Chapter 1: Linear classifiers. Updated January 19, 2024.
  • Gradient Descent Example Spreadsheet

• Neural Networks and Backpropagation. Updated January 30, 2024.
• More on Neural Net Training Updated February 12, 2024.
  • Stanford Advice on Babysitting the Learning Process Updated February 12, 2024.
• Convolutional Neural Networks Updated Feb 13, 2024
  • Pytorch Version of Chapter 3 Examples
• Recurrent Neural Networks Updated Feb 25, 2024.
  • Stanford Examples for Recurrent Neural Networks
  • Pytorch Version of Chapter 4 Examples
  • Erratum: What the book calls "backpropagation through time" (BTT) is usually called "truncated backpropagation through time" (TBTT).
• Embedding and Encoding
  • Pytorch Version of Chapter 5 Examples
• Transformers. State-of-the-art Sequence-to-Sequence Model. Updated March 12, 2024.
  • Financial Times Transformer Explanation.
• Basic Auto-Encoder. Updated March 12, 2024.
  • IPython Notebook for PCA demo
• Generative Models. Convolutional Auto-Encoder. Variational Auto-Encoder. Generative Adversarial Models. Updated March 22, 2024.
  • Pytorch Version of Chapter 7 Examples
  • Zoom Recording. Passcode = Q4Q&9A . Sorry for the AV problems on March 22, posting a recording. Starts at 25 minutes of the video.
• Markov Decision Processes. Updated April 4, 2024.
• Deep Reinforcement Learning.

Assignments

• PyTorch Introduction: PyTorch is required to complete the programming part of all assignments. We suggest these Pytorch tutorials to help you get started and also as a reference. We included a notebook in the assignment to help you get started quickly on the task.

• Assignment FAQ Answers to frequently asked questions about assignments will be updated on this page.

• Assignment 1.

  • Conceptual Exercise.
  • Programming Notebook
  • Programming Notebook pdf

• Assignment Two

  • All files. Due date: Feb 19 2024, 23:59
  • Conceptual Exercise.
  • Programming Notebook
  • Programming Notebook pdf
  • Pytorch Version of Chapter 3 Examples

• Assignment Three (Due date: Wed Mar. 11 2024, 23:59)

  • Conceptual Exercise.
  • Programming Notebook. Updated March 12, 2024
  • Programming Notebook pdf. Updated March 12, 2024.
  • Data for assignment 3
  • Bonus Question
  • Pytorch Version of Chapter 5 Examples

• Assignment Four

  • Conceptual Exercise.
  • Programming Notebook. Updated April 1, 2024.
  • Programming Notebook pdf. Updated April 1, 2024.
  • Pytorch Version of Chapter 7 Examples
  • See Training Slides for batch normalization.

Exam Information and Resources

• Final Exam Times
  • Synchronous (short answer in person) Sunday April 21 3:30-6:30 pm. ASB 9838.
  • Asynchronous (take home) April 18 1 pm - April 19 1 pm

• Practice Questions
  • Sample Exam I University of Toronto 2019
  • Sample Exam II. University of Toronto 2018. You should try to solve this first yourself, you can check your solutions here.
  • 30 questions to test a data scientist on Deep Learning
  • Canvas Practice Exam
• Final Exam Info. Updated April 17, 2024.
  • Canvas Discussion Forum Screenshot
  • Final Exam Long Answer Questions.

Learning Resources

Websites

• Stanford Course on Neural Nets for Vision.
• Gradient Descent and Adam.
• Dropout.

Demos

• Linear Classification Demo
• Other Demos

Books

• Reference book on deep learning By Bengio, Goodfellow, and Courville. Covers many topics, a good reference to get a quick idea on what a deep learning approach to a machine learning problem would be. Digital edition.

• Pattern Recognition and Machine Learning, Chris Bishop, Springer

• Pattern Classification, Duda, Hart, and Stock, Wiley. See especially "Practical Considerations for Neural Net Learning".

• Nonlinear Dimensionality Reduction