Pre-requisites | Evaluation | Logistics | Schedule | Quizzes/Assignments | What Next?

Note: The video lectures for this course are now available on youtube

Pre-requisites

---

There are no official pre-requisites for this course but it would help if you have done the following courses (preferably in the order mentioned below) :

• Calculus [Online course from MIT]

• Linear Algebra [CS6015 or equivalent] | [Online course from MIT]

• Probability Theory [CS6015 or equivalent] | [Online course from MIT]

• Non-linear Optimization [CS5020 or equivalent] | [First Course in Optimization by Prof. Soman (IITB) available on CDEEP]

• Pattern Recognition and Machine Learning [CS5691 or equivalent] | [Andrew Ng's ML course]

If you can solve most of this assignment then you are ready for this course!

Logistics

---

• Instructor: Mitesh M. Khapra

• When: Jan-May 2018

• Lectures: Slot E

• Where: Aryabhatta Hall, CS Building, First Floor

• Teaching Assistants: Purvi Goel, Raviraj Joshi, Preksha Nema, Ananya B. Sai, Soham Parikh, Gurneet Singh, Sanchit Agarwal, Shweta Bharadwaj, Mohan Bhambani, Suman Banerjee

Evaluation

---

60% Assignments | 10% Quiz 1 | 10% Quiz 2 | 20% Endsem

Schedule

---

*T = Teaching mode - this version of the slides contains animations (is good for first time viewing)
*H = Handout mode - this version of the slides will be available soon and does not contain animations (is good for revision before exams and for printing)
*M = Module (each lecture is broken down into smaller modules)

Lecture#	Contents	Lecture Slides	Lecture Videos	Extra Reading Material
Lecture 1	(Partial) History of Deep Learning, Deep Learning Success Stories	T | H	M1 | M2| M3 | M4 | M5| M6 | M7 | M8| M9	Juergen Schmidhuber, Deep Learning in Neural Networks: An Overview. 2014
Lecture 2	McCulloch Pitts Neuron, Thresholding Logic, Perceptrons, Perceptron Learning Algorithm and Convergence, Multilayer Perceptrons (MLPs), Representation Power of MLPs	T | H	M1 | M2| M3 | M4 | M5| M6 | M7 | M8	Chapters 1,2,3,4 from Neural Networks by Rojas
Lecture 3	Sigmoid Neurons, Gradient Descent, Feedforward Neural Networks, Representation Power of Feedforward Neural Networks	T | H	M1 | M2| M3 | M4 | M5	http://neuralnetworksanddeeplearning.com/chap4.html
Lecture 4	Feedforward Neural Networks, Backpropagation	T | H	M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10	See Lecture 2 [Training Neural Networks] by Hugo Larochelle
Lecture 5	Gradient Descent (GD), Momentum Based GD, Nesterov Accelerated GD, Stochastic GD, AdaGrad, RMSProp, Adam	T | H	M1 and M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M9 part 2	http://ruder.io/optimizing-gradient-descent/ http://cs231n.github.io/neural-networks-3/ AdaGrad RMSProp
Lecture 6	Eigenvalues and eigenvectors, Eigenvalue Decomposition, Basis, Principal Component Analysis and its interpretations, Singular Value Decomposition	T | H	M1 | M2| M3 | M4 | M5| M6 | M6 part 2 | M7| M8	PCA Tutorial 1 PCA tutorial 2 SVD Tutorial 1 SVD Tutorial 2
Lecture 7	Autoencoders and relation to PCA, Regularization in autoencoders, Denoising autoencoders, Sparse autoencoders, Contractive autoencoders	T | H	M1 | M2| M3 | M4 | M5| M6	Lecture 6 from Hugo Larochelle's course Denoising Autoencoders Sparse Autoencoders Contractive Autoencoders
Lecture 8	Bias Variance Tradeoff, L2 regularization, Early stopping, Dataset augmentation, Parameter sharing and tying, Injecting noise at input, Ensemble methods, Dropout	T | H	M1 | M2| M2 part 2 | M3 | M4| M5 | M6 | M7 | M8 | M9 | M10| M11	Chapter 7 from Deep Learning book Dropout Ali Ghodsi's Video Lectures on Regularization
Lecture 9	Greedy Layerwise Pre-training, Better activation functions, Better weight initialization methods, Batch Normalization	T | H	M1 | M2| M3 | M4 | M5	Batch Normalization Greedy Layerwise Pre-training Reducing the Dimensionality of Data with Networks Exploring Strategies for Training Deep Neural Networks The difficulty of training deep architectures and effect of unsupervised pre-training Why does Unsupervised Pre-Training Help?
Lecture 10	Learning Vectorial Representations Of Words	T | H	M1 | M2| M3 | M3 part 2 | M4| M5 | M5 part 2 | M6 | M7 | M8 | M9 | M10	Deriving Mikiolov's word2vec word2vec parameter learning explained Blog 1 Blog 2 Blog 3
Lecture 11	Convolutional Neural Networks, LeNet, AlexNet, ZF-Net, VGGNet, GoogLeNet, ResNet	T | H	M1 | M2| M3 | M3 part 2 | M4| M4 part 2 | M5	Chapter 9 AlexNet ZFNet VGGNet 9 Deep Learning papers GooLeNet ResNet
Lecture 12	Object Detection, RCNN, Fast RCNN, Faster RCNN, YOLO	T | H	Will be available soon	RCNN Fast RCNN Faster RCNN YOLO
Lecture 13	Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks	T | H	M1 | M2| M3 | M4 | M5| M6 | M7 | M8 | M9 | M10	RCNN(See the section on visualizations) ZFNet(see the section on experiments) Striving for simplicity(Guided Backpropagation) DeepVis Deep Inside Convolutional Networks Understanding Convolutional Networks Understanding Deep Image Representations DeepDream A Neural Algorithm for Artistic Style Neural style Intriguing properties of neural networks Deep Neural Networks are easily fooled
Lecture 14	Recurrent Neural Networks, Backpropagation Through Time (BPTT), Vanishing and Exploding Gradients, Truncated BPTT	T | H	M1 | M2| M3 | M4 | M5	On the difficulty of training RNNs
Lecture 15	Gated Recurrent Units (GRUs), Long Short Term Memory (LSTM) Cells, Solving the vanidhing gradient problem with LSTMs	T | H	M1 | M2| M3 | M3 part 2	LSTM: A Search Space Odyssey Understanding Deriving and Extending the LSTM Understanding LSTM Networks (Nice blog) Paper which introduced Gated Recurrent Units Paper which introduced LSTMs
Lecture 16	Encoder Decoder Models, Attention Mechanism, Attention over images, Hierarchical Attention	T | H	M1 | M2| M3 | M3 part 2 | M4| M5	Attention-based Encoder–Decoder Networks
Lecture 17	Directed Graphical Models	T | H	Will be available soon	Graphical Models in a Nutshell
Lecture 18	Markov Networks	T | H	Will be available soon
Lecture 19	Using joint distributions for classification and sampling, Latent Variables, Restricted Boltzmann Machines, Unsupervised Learning, Motivation for Sampling	T | H	Will be available soon	An Introduction to Restricted Boltzmann Machines
Lecture 20	Markov Chains, Gibbs Sampling for training RBMs, Contrastive Divergence for training RBMs	T | H	Will be available soon	A Practical Guide to Training RBMs Training RBMs
Lecture 21	Variational autoencoders	T | H	Will be available soon	Tutorial on Variational Autoencoders by Carl Doersch Blog by Jaan Altosaar
Lecture 22	Autoregressive Models: NADE, MADE, PixelRNN	T | H	Will be available soon	NADE paper MADE paper PixelRNN paper
Lecture 23	Generative Adversarial Networks (GANs)	T | H	Will be available soon

Quizzes/Assigments

---

	Topics	Release Date	Submission Date
Assignment 1	Derivatives + Probability [Theory]	16-Jan-2018	23-Jan-2018
Assignment 2	Linear Algebra [Theory]	23-Jan-2018	30-Jan-2018
Quiz 1	Lectures 1-7	15-Feb-2018
Assignment 3	Backpropagation [Programming]	30-Jan-2018	14-Feb-2018
Assignment 4	Word2Vec [Programming]	14-Feb-2018	28-Feb-2018
Assignment 5	Convolutional Neural Networks [Programming]	28-Feb-2018	16-Mar-2018
Assignment 6	Probability Refresher [Theory]	14-Mar-2018	23-Mar-2018
Assignment 7	Recurrent Neural Networks [Programming]	23-Mar-2018	11-Apr-2018
Quiz II	Lectures 8-15	04-Apr-2018
Assignment 8	Restricted Boltzmann Machines [Programming]	11-Apr-2018	26-Apr-2018
End Sem	Lectures 1-23	04-May-2018

What Next?

---

You may find the following courses to be useful:

• Deep Learning for Computer Vision [from Stanford]

• Deep Learning for NLP [from Stanford]