Here are some suggestions for course projects. Feel free to suggest your own. See the section on Additional Readings. You should discuss the topic with me before you start in any case.

The typical project would evaluate a deep learning method on an interesting problem of your choice. "Interesting" means some mix of real-world importance and learning complexity. I assume that many students will already have a learning problem that they are working on, so it is natural to try deep learning methods. I'm also open to other types of projects, for instance surveys or theoretical analysis. Below I list example topics.

Compare with other models for learning latent features

1. PCA,  kernelized PCA (non-linear).
2. Neural nets.
3. Hierarchical Clustering, Multiple Clustering.
4. SVMs: can we think of deep learning as learning a kernel? How do deep models compare to SVMs with latent variables?
   • Consider the experiments and arguments in "Scaling Learning Towards AI".
     • How does a Deep Belief net perform in learning parity? The sine function?
     • The paper reports a dataset where a convolutional neural net performs much better than an SVM with a local kernel. How about a DBN vs. SVM? Can you find a dataset with the reverse performance, where an SVM outperforms a DBN? If not, why not? What would such a dataset be like?

Compare Learning Methods for latent feature models

1. EM
2. gradient descent, e.g. conjugate version.

Compare different deep learning Models

1. Number of layers.
2. Different types of models, e.g.
   1. Logistic Regression
   2. Multilayer perceptron
   3. Deep Convolutional Network
   4. Auto Encoders, Denoising Autoencoders
   5. Stacked Denoising Auto-Encoders
   6. Restricted Boltzmann Machines
   7. Deep Belief Networks

Apply Deep Learning

1. NLP
2. Vision
3. Multimodal (see tutorials)
4. Transfer Learning.
   • Do DBNs achieve the invariance and feature abstraction discussed in the paper? Consider this type of experiment: first learn a DBN that represents the sine function. Then use the weight settings as initial settings for learning the cosine function.

Surveys

• Literature Review
• Theoretical Analysis
  • We discussed the equivalence between regularlized neural net learning with 1 hidden layer vs. support vector machine learning. Is it possible to extend this to deep neural nets with several hidden layers?