Not logged in. Login

Deep Learning - CMPT 880 G1

Spring Semester 2019

Simon Fraser University

Instructor: Oliver Schulte

Breadth Area III

Course Logistics

Office Location: TASC 1 9021.
Office Phone: 778-782-3390.
Office Hours: Thursday 2 pm - 3 pm.
E-mail Office Hour: Friday 9:30 am - 10:30 am.
TA: Xia Hu (huxiah@sfudotca)
TA Office Hour: Tuesday 11 am - 12 am, ASB 9810.
Email: myfirstname_mylastname@sfudotca


  • Feb 28, by Xia Hu: 'Interpretability of machine learning.' Slides
  • final project presentations during the exam period on Monday April 15 from 2:30-6:30 pm, in TASC 1 9204. There will be a coffee break. Information on Projects. Todo items:
    • Please indicate next to your project outline presentation whether your group wants to present a poster or an oral presentation. If I don't hear from you I assume you prefer a poster presentation (default = poster).
    • Please remember to post an electronic copy of your poster/presentation by the evening of April 15.

Course Information

Plan for the Course

The course has three main parts.

  1. Intro to Neural nets. Depending on the students' background, this will take about 1/3 of the course.
    • Definition of feed-forward neural nets, and related machine learning concepts (e.g. linear classifiers, k-nearest neighbour).
    • Training: basic back propagation and refinements.
    • Definition of deep feed-forward neural nets. Training: back propagation + special techniques (e.g. drop-out, rectified linear unit, norm constraint).
    • Convolutional Neural Nets for vision (and sequences)
    • Autoencoders for learning features and embeddings. Related machine learning models (e.g. clustering, self-organizing maps, principal component analysis).
    • Recurrent and long short term memory networks for sequential data and processing. Related machine learning models (e.g. hidden Markov models).
  2. Advanced Topics. Advanced topics selected by groups of students depending on which they want to discuss. Depending on student interest, this will take about 1/3 of the class. The schedule below lists a set of topics to choose from.
  3. Applications and Projects. The last third of the course is geared towards applications, trying out deep learning on real problems.

Information for Topic Presentations

This is a seminar course, so students are expected to cover part of the course material in presentations and discussions. Every student will contribute one presentation about a deep learning topic. You can choose from a list as shown on the course schedule. Information on topic presentations.

You have the option of presenting as a group, and I strongly encourage you to do so, for the following reasons.

  1. Discussing topics with other students will help you understand them.
  2. Having help with the presentation will lead to a better presentation and a better grade.
  3. You can build on the group presentation for your course project.

Information for Projects

You should do projects in groups, with at least two and at most 5 members. Each project group will contribute two presentations.

  1. A project outline presentation for your course project: introduce the problem, the dataset, describe your plan of attack. Information on Project Outlines.
  2. A final project presentation about the results of your course project. This will be part of a final project day during the final exam period. Information for Projects.

Course Schedule

  • Week 1 (January 3). Introduction.
  • Week 2 (January 8 and 10).
  • k-nearest Neighbour Classification. Slides. Image Classification. Updated January 8, 2019
  • Linear Classification.
  • Week 3 (January 15): Linear Classification.
  • Week 4 (January 22 and 24).
    • Neural Nets and Backpropagation.
  • Week 5 (January 29 and 31): Deep Neural Networks.
    • Backpropagation Exercises due on February 5.
    • Training deep neural networks.

Part II: Special network architectures. Summary Slides.

  • Part III: Week 10-13 (= March 14-April 2). Student Presentations. Students should present in groups of 3-4. A good resource for presentation material is the official deep learning site. Each student should contribute two types of presentations.
  • Week 11: Reinforcement Learning.
  • Week 13 (April 4): no class. Instead we will have a long meeting with final project presentations during the exam period on Monday April 15 from 2:30-6:30 pm. There will be a coffee break. Information on Projects
    • Project Support Session April 4. Attendance is optional, I'll be in the regular class room, if you want to discuss your project in addition to the office hour.

Backpropagation Exercises

  1. The derivation slides for backpropagation state that "Proof by backward induction. Easy to see that the claim is true for output nodes. (Exercise).". Write down formally the claim for output nodes and prove it.
  2. Trace backpropagation.
  3. Backpropagation for different error functions. Consider the back propagation algorithm from the class slides. This implements gradient descent for the squared error loss. How would you have to change the back propagation algorithm so that it implements gradient descent for the following error functions? Briefly justify your answer.
    1. The perceptron error function.
    2. The cross-entropy error used in logistic regression.


You should submit output (about one page) that verifies that you can train a feed-forward single-layer neural net on the MNIST dataset. To give you an idea of the performance you can expect, the vanilla implementation in the deep learning tutorial gets a test set error of 1.65% after training for about 100 min. This exercise is not about maximizing performance but if you are too far from this it's a sign you are not using your neural net learning system correctly.

Presentation Schedule

The length of one presentation should be about 15 minutes, which allows for up to 10 minutes of questions afterwards. Please sign up for a presentation as soon as possible so we can resolve conflicts. You can edit this web page. An example is given below.

Your Names. Topic.

  1. Ricardo Silva Carvalho, Muhammad Raihan Muhaimin; Topic: Optimization: Momentum, RMSProp, Adam and one cycle policy. Date: February 12. Final Presentation Type: In class presentation. Slides
  2. Kumar Abhishek and Nishant Kambhatla, Attention Models, February 12, 2019. [ Slides ]
  3. Yiqi Yan, Jiansheng Ding, Xudong Luo, Mohan Zhang. Generative adversarial networks basis + Progressive Growing of GANs. February 12, 2019. Slides
  4. Ankita Sakhuja, Kiarash Zahirnia; Topic: Network Embedding and Heterogeneous Representation Learning. Date: February 26,2019. Slides
  5. Jillian Anderson, Anna Mkyrtchyn, Brie Hoffman. Long Short-Term Memory Networks. March 5, 2019. Slides
  6. Kinnan Kwok. Variational Autoencoders. March 5, 2019. Slides
  7. Shubam Sachdeva, Seerat Randhawa. Principal Component Analysis. March 5, 2019. Slides
  8. Hamid Homapour, Amirhossein Mozafari, Farzad Sharif, March 12, 2019; Neural Turing Machines.
  9. Darren Sutton, Lucas Wu. March 14, 2019. Adversarial Attacks and Interpretability. Slides
  10. Jarrod Haas. March 26, 2019. Normalizing Flows.
  11. Nazanin Mehrasa, Akbar Rafiey, Akash Abdu Jyothi, Sha Hu. March 28, 2019. VAE for Temporal Point Process. Slides
  12. Morgan Heisler and Julian Lo. April 2nd, 2019. Recommender Networks Slides


Final Project Outline Presentations

  1. Ricardo Silva Carvalho, Muhammad Raihan Muhaimin, Ankita Sakhuja, Kiarash Zahirnia. March 21, 2019. Public Daycare Detection on Satellite Imagery. (Oral Presentation)Slides for Outline Slides for Final
  2. Morgan Heisler and Julian Lo. March 21, 2019. Optic Nerve Head Parameterization Using Deep Neural Networks. Slides (Oral Presentation)
  3. Jillian Anderson, Anna Mkyrtchyn, Brie Hoffman. March 26, 2019. Predicting News Item Popularity Slides
  4. Nazanin Mehrasa, Akbar Rafiey, Akash Abdu Jyothi, Sha Hu, Hamid Homapour. March 28, 2019. VAE for Temporal Point Process.Slides (Oral presentation)
  5. Kumar Abhishek and Nishant Kambhatla, March 28, 2019. Exploring the Generalizability of Sequence-to-Sequence Architectures. [ Project Outline ], [ Final Presentation ] [Oral Presentation]
  6. Jiansheng Ding, Xudong Luo, Yiqi Yan, Mohan Zhang. March 28, 2019. Attention Based CNN for Skin Lesion Diagnosis. Poster
  7. Darren Sutton and Lucas Wu. TBD. April 2, 2019.
  8. Jarrod Haas and Kinnan Kwok. Synthetic Data Generation for Image Segmentation. April 2, 2019. (Presentation)
  9. Shubam Sachdeva and Seerat Randhawa, Video Inpainting for Arbitrary Foreground Object Removal, April 2, 2019.(Presentation)
  10. Farzad Sharif Bakhtiar, Posteriori Feature Attention, April 4, 2019. Project Outline Presentation Slides - Poster Presentation at final session.



  • Recent book on deep learning. By Bengio, Goodfellow, and Courville. Covers many topics, a good reference to get a quick idea on what a deep learning approach to a machine learning problem would be.
  • Pattern Recognition and Machine Learning, Chris Bishop, Springer
  • Pattern Classification, Duda, Hart, and Stock, Wiley. See especially "Practical Considerations for Neural Net Learning".
Updated Mon April 15 2019, 23:13 by kabhishe.