CMPT 882: Deep Neural Networks.

Simon Fraser University, Burnaby Campus. Fall 2018.

Tuesday and Thursday 2:30-3:50pm. Room: RCB 8100 (Tuesday), C 9000 (Thursday)

Announcements

• 2018-09-01: Welcome to CMPT 882!

Instructors

Martin Ester

• Web page
• Office: TASC 1 9231
• Email: ester at sfu dot ca

Maxwell Libbrecht

• Web page
• Office: TASC 1 9219
• Email: maxwl at sfu dot ca

Course information

This course will introduce students to neural networks and their applications. We will aim to cover methods that apply a variety of network architectures, learning strategies and implementations. This will be a seminar course, where students will present selected research papers and conduct small research projects. An emphasis of the course will be on building communication skills: writing and giving presentations. General guidelines and strategies for writing clearly and giving good talks will be given, and students will receive feedback on their presentations and project reports from the instructor and other students. Application areas include: Vision Speech Natural language Genomics Medical imaging.

• Grading: 30% talk, 50% project, 10% committee participation, 10% class participation.

Schedule

Schedule

Lecture slides

• CNNs, RNNS, and LSTMs
• AEs and VAEs
• How to write a good paper
• DeepSEA slides
• avocado

Presentation

Each student will give a talk on a recent paper. The talks will be 30 minutes plus 5 minutes for questions.

• Peer feedback: Each speaker will get a committee of three other students, who will help the speaker improve their talk. The committee will have one primary member who will read the paper and attend the practice talk and two secondary members who will just attend the practice talk.
• Deliverables (upload all deliverables to Google Drive, as described below):
  • [Due 14 days before talk, by speaker] Choose paper. Write 1 page summarizing paper and why it is relevant for the class.
  • [Due 7 days before talk, by primary committee member] Primary reads paper. Speaker and primary meet 7-14 days before talk, discuss paper and slides. Primary writes 1 page summarizing discussion and giving feedback to primary. Speaker submits draft slides.
  • [Due 5 days before talk, by all committee members] Speaker gives practice talk to all three committee members (it's up to the speaker to reserve a room). Each committee member fills out an evaluation form.
• How to choose a paper:
  • A paper should (1) involve machine learning in a life sciences application (genomics, imaging, medicine, etc); (2) be published in the last three years; and (3) be of interest to the class, meaning that it is understandable to a computer scientist without knowledge of the application area and the methods are a primary focus of the paper.

Evaluation sheet

Course project

Students will conduct original research projects on a topic related to the course. The deliverable of the course project is a report to be written in the style of a research paper. We have broken down the process of writing the report into three steps with two intermediate versions of the report to be submitted, as specified below. Groups of up to three students are allowed. You must specify defined responsibilities for each group member in the proposal.

To clarify the expectations for projects, here is a short description of the most common types of course projects with their corresponding expectations:

• Benchmarking of existing algorithms on a new, interesting dataset: Use a dataset that is of interest to you and that you know well enough to be able to interpret your results. Choose at least three state-of-the-art algorithms and thoroughly evaluate their performance on your dataset. Discuss your findings, such as why some algorithm outperformed the others or weaknesses of the algorithms that suggest directions for future research.
• Extension of an existing algorithm: Identify a weakness of an existing, state-of-the art algorithm, such as inaccuracy or inefficiency. Design and implement an extension of the algorithm that overcomes that weakness and compare it on some dataset against the original algorithm. The extension should have some technical novelty.
• Proposal of a new algorithm: This is the most creative and most open-ended of the project types listed here. This type of project designs and implements a new algorithm for a known problem or a variant of such a problem. The algorithm should be experimentally compared against one state-of-the-art algorithm for that problem.

Deliverables:

• [Due EOD Monday Oct 29] Proposal. Project proposals should be approximately three pages and have the following sections: (1) Introduction/Motivation, (2) Previous work, (3) Planned methods, (4) Anticipated results, (5) Timeline. The proposal should also include one figure giving a graphical depiction of the planned project. The timeline should specify what work will be finished by the milestone and, for group projects, the group member responsibilities.
• [Due EOD Monday Nov 19] Milestone. Five pages with the same sections as the proposal, but with "Methods" instead of planned methods.
• [Due EOD Monday Dec 10] Final version of project paper. Same sections, but with "Methods" and "Results" instead of planned and anticipated. Ten pages.

How to submit deliverables

Google Drive submission directory

• Open the Google Drive submission directory, linked above.
• Log in to your Google account. If you don't have one already, you can create one for free.
• Click on the folder corresponding to the *speaker* this deliverable corresponds to. (If you are the speaker, go to your own directory; if you are a committee member, go to the speaker's directory.) If the directory does not exist yet, create it using the New button in the top left.
• Upload your deliverable with the following name: "<your_name> - <deliverable_name>.<extension>". For text deliverables, you can use a googledoc rather than a file from your computer. Deliverable names: "Speaker paper summary"; "Speaker draft slides"; "Primary draft slide feedback"; "Practice talk feedback".
• For example, if my name is Adam, I am Bob's primary and I am submitting the feedback from the one-on-one draft slides meeting, I would name my file "Adam - Primary draft slide feedback.txt" and put it in the directory named "Bob".
• Put project deliverables in your own folder, using the names: "Project proposal"; "Project milestone"; "Project writeup". For group projects, choose one group member and put deliverables in their directory. Other group members, add an (empty) file named "Group member - <name>" to your directory, where <name> is the folder with the deliverables.

Talk study questions

Committee members: Use these questions to help give feedback on a speaker's practice talk and outline.

• Slides:
  • Is each slide headed by a full sentence expressing the take-home message of that slide?
  • Are the graphics and font (including figure axes and legends) on each slide large enough to be legible?
  • Has all extraneous information been removed from each slide?
  • Do each slide's elements appear as they are needed by the speaker?
• Organization:
  • Does the talk have a home slide/image that helps the audience reorient themselves?
  • Is the talk separated into distinct episodes or "data dives" that are independently understandable?
• Introduction:
  • Is the talk of interest to a general CS/ML audience?
  • Does the introduction gain attention and interest?
  • Does the introduction give sufficient background for a nonspecialist audience?
• Content:
  • Is the level of detail appropriate?
  • Is the amount of content appropriate for a 40 minute talk?
  • Notation: Is mathematical notation defined clearly, and redefined with each use?
  • Does the talk focus on the key points of the paper?
• Analysis:
  • Does the talk put the paper in context of previous work?
  • Does the talk investigate not just *what* the authors did but *why* they did it?
  • Does the speaker critically analyze the correctness and impact of the paper?
  • Does the speaker explain the paper's implications, caveats and future work?
• Speech:
  • Is the speaker in control of his/her voice? Consider enunciation, projection (loud/soft), rate of speech, use of pauses and presence of verbal ticks ("um", "like").
  • Does the speaker have appropriate rapport with the audience? Consider eye contact, body language.

Background reading

Here are some materials if you want to brush up on your deep learning background:

• Very short introductory blog post
• One-day deep learning workshop at MIT
• Deep learning book by Ian Goodfellow, Yoshua Bengio and Aaron Courville.
• Deep learning coding tutorials using Theano
• Deep learning course from Coursera

Paper suggestions

• Finnegan A, Song JS. Maximum entropy methods for extracting the learned features of deep neural networks. PLoS computational biology. 2017 Oct 30;13(10):e1005836.
• Chidester B, Do MN, Ma J. Rotation Equivariance and Invariance in Convolutional Neural Networks. arXiv preprint arXiv:1805.12301. 2018 May 31.
• Koh PW, Pierson E, Kundaje A. Denoising genome-wide histone ChIP-seq with convolutional neural networks. Bioinformatics. 2017 Jul 12;33(14):i225-33.
• Shrikumar A, Su J, Kundaje A. Computationally Efficient Measures of Internal Neuron Importance. arXiv preprint arXiv:1807.09946. 2018 Jul 26.
• Shrikumar A, Greenside P, Kundaje A. Learning important features through propagating activation differences. arXiv preprint arXiv:1704.02685. 2017 Apr 10.
• Semi and Weakly Supervised Semantic Segmentation Using Generative Adversarial Network. https://arxiv.org/pdf/1703.09695.pdf Computational biology
• Basset: Learning the regulatory code of the accessible genome with deep convolutional neural networks. David R Kelley, Jasper Snoek and John Rinn.
• DFIM: Deep Feature Interaction Maps uncover latent dependence structure encoded in deep learning models of regulatory DNA sequences. Greenside PG, Shimko T, Fordyce P, Kundaje A.
• Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts. Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, Ed Chi, Proceedings ACM SIGKDD 2018
• Towards Explanation of DNN-based Prediction with Guided Feature Inversion. Mengnan Du, Ninghao Liu, Qingquan Song, Xia Hu, Proceedings ACM SIGKDD 2018
• Learning Deep Network Representations with Adversarially Regularized Autoencoders. Wenchao Yu, Cheng Zheng, Wei Cheng, Charu Aggarwal, Dongjin Song, Bo Zong, Haifeng Chen, Wei Wang, Proceedings ACM SIGKDD 2018
• NetWalk: A Flexible Deep Embedding Approach for Anomaly Detection in Dynamic Networks. Wenchao Yu, Wei Cheng, Charu Aggarwal, Kai Zhang, Haifeng Chen, Wei Wang, Proceedings ACM SIGKDD 2018
• Deep Variational Network Embedding in Wasserstein Space. Dingyuan Zhu, Peng Cui, Daixin Wang, Wenwu Zhu, Proceedings ACM SIGKDD 2018
• Adversarial Attacks on Neural Networks for Graph Data. Daniel Zügner, Amir Akbarnejad, Stephan Günnemann, Proceedings ACM SIGKDD 2018

Groups that work on deep learning:

• Geoffrey Hinton (methods)
• DeepMind
• Google Brain
• Allen Institute for Artificial Intelligence (NLP)
• Brendan Frey (computational biology)
• Anshul Kundaje (computational biology)

Conferences/journals:

• Neural Information Processing Systems (NIPS)
• International Conference on Machine Learning (ICML)
• International Conference on Learning Representations (ICLR)
• Journal of Machine Learning Research (JMLR)