Spring Semester 2019

Simon Fraser University

Instructor: Oliver Schulte

Breadth Area III

Course Logistics

Office Location: TASC 1 9021.
Office Phone: 778-782-3390.
Office Hours: Thursday 2 pm - 3 pm.
E-mail Office Hour: Friday 9:30 am - 10:30 am.
TA: Xia Hu (huxiah@sfudotca)
TA Office Hour: Tuesday 11 am - 12 am, ASB 9810.
Email: myfirstname_mylastname@sfudotca

Announcements

• Feb 28, by Xia Hu: 'Interpretability of machine learning.' Slides
• final project presentations during the exam period on Monday April 15 from 2:30-6:30 pm, in TASC 1 9204. There will be a coffee break. Information on Projects. Todo items:
  • Please indicate next to your project outline presentation whether your group wants to present a poster or an oral presentation. If I don't hear from you I assume you prefer a poster presentation (default = poster).
  • Please remember to post an electronic copy of your poster/presentation by the evening of April 15.

Course Information

• Syllabus. Updated January 1, 2019.
• Resources: In addition to my slides we will make use of the following.
  • Stanford Course on Neural Nets for Vision.
  • On-line deep learning reference book.

Plan for the Course

The course has three main parts.

1. Intro to Neural nets. Depending on the students' background, this will take about 1/3 of the course.
   • Definition of feed-forward neural nets, and related machine learning concepts (e.g. linear classifiers, k-nearest neighbour).
   • Training: basic back propagation and refinements.
   • Definition of deep feed-forward neural nets. Training: back propagation + special techniques (e.g. drop-out, rectified linear unit, norm constraint).
   • Convolutional Neural Nets for vision (and sequences)
   • Autoencoders for learning features and embeddings. Related machine learning models (e.g. clustering, self-organizing maps, principal component analysis).
   • Recurrent and long short term memory networks for sequential data and processing. Related machine learning models (e.g. hidden Markov models).
2. Advanced Topics. Advanced topics selected by groups of students depending on which they want to discuss. Depending on student interest, this will take about 1/3 of the class. The schedule below lists a set of topics to choose from.
3. Applications and Projects. The last third of the course is geared towards applications, trying out deep learning on real problems.

Information for Topic Presentations

This is a seminar course, so students are expected to cover part of the course material in presentations and discussions. Every student will contribute one presentation about a deep learning topic. You can choose from a list as shown on the course schedule. Information on topic presentations.

You have the option of presenting as a group, and I strongly encourage you to do so, for the following reasons.

1. Discussing topics with other students will help you understand them.
2. Having help with the presentation will lead to a better presentation and a better grade.
3. You can build on the group presentation for your course project.

Information for Projects

You should do projects in groups, with at least two and at most 5 members. Each project group will contribute two presentations.

1. A project outline presentation for your course project: introduce the problem, the dataset, describe your plan of attack. Information on Project Outlines.
2. A final project presentation about the results of your course project. This will be part of a final project day during the final exam period. Information for Projects.

Course Schedule

• Week 1 (January 3). Introduction.
• Week 2 (January 8 and 10).
• k-nearest Neighbour Classification. Slides. Image Classification. Updated January 8, 2019
• Linear Classification.
  • Slides. Updated Jan 10, 2019.
  • Image Linear Classification.
  • Optimization and Gradient Descent. Image Classification Optimization.
• Week 3 (January 15): Linear Classification.
  • No class on January 17
  • In-class On-line background quiz in-class on January 15
  • Stanford Intro Slides
  • Demos
• Week 4 (January 22 and 24).
  • Neural Nets and Backpropagation.
    • My Slides. Updated January 30, 2018. Changed the back propagation picture to be consistent with the notation in the slides. Also added 1/2 to all expressions for the squared-loss error function.
      • Stanford Slides on Training for Images.
• Week 5 (January 29 and 31): Deep Neural Networks.
  • Backpropagation Exercises due on February 5.
  • Training deep neural networks.

• Week 6 (February 5 and 7):
  • Neural Nets and Machine Learning.
    • The Equivalence of Support Vector Machine and Regularization Neural Networks. Shows a systematic translation between a support vector machine and a neural net with one hidden layer. Lots of math, concise (only 6 pages). Here's my very rough summary.
    • The Bias-Variance trade-off. Slides
  • Homework 1 due (installing a neural net package) on February 8.

• Week 7 (February 12 and 14): Training Neural Nets: Advanced Techniques. Slides on Advanced Techniques. More on Training for Images.
  • Dropout. Presented by instructor.
  • Batch Normalization. Optional topic, can be selected for presentation by students.
  • Adapting Step Sizes (ADAM). Optional topic, can be selected for presentation by students.

Part II: Special network architectures. Summary Slides.

• Week 8 (February 26 and 28).
  • Convolutional Neural Nets for Vision. Optional topic, can be selected for presentation by students.
    • My CNN Slides
    • Stanford Intro
    • CNN IPython Notebook

• Week 9 (March 5 and 7).
  • Feature Discovery. Optional topic, can be selected for presentation by students.
    • Principal Component Analysis.
    • Autoencoders and Embeddings.
  • Homework 2 due on March 5 (competition on a data set).
  • Sequential Models I: Recurrent Neural Networks. Optional topic, can be selected for presentation by students.
  • The Deep Learning reference book has a detailed overview of different models, their strengths and weaknesses, and many references.
  • Stanford Intro
  • Hidden Markov Model Demo.

• Week 10 (March 12). Sequential Models II: Long Short-Term Memory Networks. Optional topic, can be selected for presentation by students.
  • Stanford Intro.
  • LSTM diagram.
  • LSTM Visualizer

• Part III: Week 10-13 (= March 14-April 2). Student Presentations. Students should present in groups of 3-4. A good resource for presentation material is the official deep learning site. Each student should contribute two types of presentations.
  • Present an advanced topic. Information on topic presentations.
  • Present an outline for a final project. Information on Project Outlines.
• Week 11: Reinforcement Learning.
• Week 13 (April 4): no class. Instead we will have a long meeting with final project presentations during the exam period on Monday April 15 from 2:30-6:30 pm. There will be a coffee break. Information on Projects
  • Project Support Session April 4. Attendance is optional, I'll be in the regular class room, if you want to discuss your project in addition to the office hour.

Backpropagation Exercises

1. The derivation slides for backpropagation state that "Proof by backward induction. Easy to see that the claim is true for output nodes. (Exercise).". Write down formally the claim for output nodes and prove it.
2. Trace backpropagation.
3. Backpropagation for different error functions. Consider the back propagation algorithm from the class slides. This implements gradient descent for the squared error loss. How would you have to change the back propagation algorithm so that it implements gradient descent for the following error functions? Briefly justify your answer.
   1. The perceptron error function.
   2. The cross-entropy error used in logistic regression.

HomeWorks

• Homework 1: Install Deep Learning Code. My preference would be for everyone to use the same Theano code as in the Deep Learning Tutorial. But I'm willing to accommodate preferences for different programming languages. I've created a discussion forum where you can post and (I hope) answer questions about installations. A variety of other systems are described on the deep learning site, as well as in the Stanford course notes including the following.
  • Keras
  • Caffe
  • Theano.
  • cuda-covnet
  • Torch
  • Matlab
  • Tensor Flow

You should submit output (about one page) that verifies that you can train a feed-forward single-layer neural net on the MNIST dataset. To give you an idea of the performance you can expect, the vanilla implementation in the deep learning tutorial gets a test set error of 1.65% after training for about 100 min. This exercise is not about maximizing performance but if you are too far from this it's a sign you are not using your neural net learning system correctly.

• Homework 2: Train a competitive neural net for a given data set.
  • https://coursys.sfu.ca/2019sp-cmpt-880-g1/pages/Homework2_Description
  • Data: https://coursys.sfu.ca/2019sp-cmpt-880-g1/pages/Homework2_data.zip
  • Homework2-hints

Presentation Schedule

The length of one presentation should be about 15 minutes, which allows for up to 10 minutes of questions afterwards. Please sign up for a presentation as soon as possible so we can resolve conflicts. You can edit this web page. An example is given below.

Your Names. Topic.

1. Ricardo Silva Carvalho, Muhammad Raihan Muhaimin; Topic: Optimization: Momentum, RMSProp, Adam and one cycle policy. Date: February 12. Final Presentation Type: In class presentation. Slides
2. Kumar Abhishek and Nishant Kambhatla, Attention Models, February 12, 2019. [ Slides ]
3. Yiqi Yan, Jiansheng Ding, Xudong Luo, Mohan Zhang. Generative adversarial networks basis + Progressive Growing of GANs. February 12, 2019. Slides
4. Ankita Sakhuja, Kiarash Zahirnia; Topic: Network Embedding and Heterogeneous Representation Learning. Date: February 26,2019. Slides
5. Jillian Anderson, Anna Mkyrtchyn, Brie Hoffman. Long Short-Term Memory Networks. March 5, 2019. Slides
6. Kinnan Kwok. Variational Autoencoders. March 5, 2019. Slides
7. Shubam Sachdeva, Seerat Randhawa. Principal Component Analysis. March 5, 2019. Slides
8. Hamid Homapour, Amirhossein Mozafari, Farzad Sharif, March 12, 2019; Neural Turing Machines.
9. Darren Sutton, Lucas Wu. March 14, 2019. Adversarial Attacks and Interpretability. Slides
10. Jarrod Haas. March 26, 2019. Normalizing Flows.
11. Nazanin Mehrasa, Akbar Rafiey, Akash Abdu Jyothi, Sha Hu. March 28, 2019. VAE for Temporal Point Process. Slides
12. Morgan Heisler and Julian Lo. April 2nd, 2019. Recommender Networks Slides

Presentation

Final Project Outline Presentations

1. Ricardo Silva Carvalho, Muhammad Raihan Muhaimin, Ankita Sakhuja, Kiarash Zahirnia. March 21, 2019. Public Daycare Detection on Satellite Imagery. (Oral Presentation)Slides for Outline Slides for Final
2. Morgan Heisler and Julian Lo. March 21, 2019. Optic Nerve Head Parameterization Using Deep Neural Networks. Slides (Oral Presentation)
3. Jillian Anderson, Anna Mkyrtchyn, Brie Hoffman. March 26, 2019. Predicting News Item Popularity Slides
4. Nazanin Mehrasa, Akbar Rafiey, Akash Abdu Jyothi, Sha Hu, Hamid Homapour. March 28, 2019. VAE for Temporal Point Process.Slides (Oral presentation)
5. Kumar Abhishek and Nishant Kambhatla, March 28, 2019. Exploring the Generalizability of Sequence-to-Sequence Architectures. [ Project Outline ], [ Final Presentation ] [Oral Presentation]
6. Jiansheng Ding, Xudong Luo, Yiqi Yan, Mohan Zhang. March 28, 2019. Attention Based CNN for Skin Lesion Diagnosis. Poster
7. Darren Sutton and Lucas Wu. TBD. April 2, 2019.
8. Jarrod Haas and Kinnan Kwok. Synthetic Data Generation for Image Segmentation. April 2, 2019. (Presentation)
9. Shubam Sachdeva and Seerat Randhawa, Video Inpainting for Arbitrary Foreground Object Removal, April 2, 2019.(Presentation)
10. Farzad Sharif Bakhtiar, Posteriori Feature Attention, April 4, 2019. Project Outline Presentation Slides - Poster Presentation at final session.

Resources

• Deep Learning Website

• Deep Learning Software, including a Matlab Toolbox

• Nonlinear Dimensionality Reduction

Books

• Recent book on deep learning. By Bengio, Goodfellow, and Courville. Covers many topics, a good reference to get a quick idea on what a deep learning approach to a machine learning problem would be.

• Pattern Recognition and Machine Learning, Chris Bishop, Springer

• Pattern Classification, Duda, Hart, and Stock, Wiley. See especially "Practical Considerations for Neural Net Learning".