CS6910/CS7015: Deep Learning

Note: The video lectures for this course are now available on youtube

Pre-requisites

There are no official pre-requisites for this course but it would help if you have done the following courses (preferably in the order mentioned below) :

Calculus [Online course from MIT]
Linear Algebra [CS6015 or equivalent] | [Online course from MIT]
Probability Theory [CS6015 or equivalent] | [Online course from MIT]
Non-linear Optimization [CS5020 or equivalent] | [First Course in Optimization by Prof. Soman (IITB) available on CDEEP]
Pattern Recognition and Machine Learning [CS5691 or equivalent] | [Andrew Ng's ML course]

If you can solve most of this assignment then you are ready for this course!

Logistics

Instructor: Mitesh M. Khapra
When: Jan-May 2024
Lectures: Slot H
Where: CS25
Teaching Assistants:

Name	Lab	Email	Office hours	Days
Anushka Singh	AI4Bharat	cs22s015@smail.iitm.ac.in	3-4 pm	Tuesday, Thursday
Oikantik Nath	AI4Bharat	cs22s013@smail.iitm.ac.in	12-1 pm	Tuesday, Thursday
Bibhuti Majhi	AI4Bharat	cs22m031@smail.iitm.ac.in	2-4 pm	Wednesday
Sarthak Naithani	AI4Bharat	cs22m078@smail.iitm.ac.in	3-4 pm	Tuesday, Wednesday
Putta Sai Sree Ram	AI4Bharat	cs22m076@smail.iitm.ac.in	3-4 pm	Monday, Friday
Ravi Prakash Singh	AI4Bharat	cs22m069@smail.iitm.ac.in	1-2 pm	Tuesday, Friday
Guddeppagari Shathish Kumar Reddy	AI4Bharat	cs22m080@smail.iitm.ac.in	3-4 pm	Monday, Friday
Poorbi Mukesh Dalal	-	cs22m064@smail.iitm.ac.in	1-3 pm	Wednesday
Ashok R	-	cs22m023@smail.iitm.ac.in	2-4 pm	Wednesday
Amit Kumar	-	cs22m010@smail.iitm.ac.in	2-4 pm	Wednesday

Evaluation

40% Assignments (20% each assignment) | 10% Project Phase 1 | 20% Project Phase 2 | 30% Endsem

Reference Textbooks

Ian Goodfellow and Yoshua Bengio and Aaron Courville. Deep Learning. An MIT Press book. 2016.
Charu C. Aggarwal. Neural Networks and Deep Learning: A Textbook. Springer. 2019.
Dive into Deep Learning

Schedule

*M = Module (each lecture is broken down into smaller modules)

Lecture#	Contents	Lecture pdf	Lecture Videos	Extra Reading Material
Lecture 1	(Partial) History of Deep Learning, Deep Learning Success Stories	Slides	M1 \| M2\| M3 \| M4 \| M5	Juergen Schmidhuber, Deep Learning in Neural Networks: An Overview. 2014
Lecture 2	McCulloch Pitts Neuron, Thresholding Logic, Perceptrons, Perceptron Learning Algorithm and Convergence, Multilayer Perceptrons (MLPs), Representation Power of MLPs	Slides	M1 \| M2\| M3 \| M4 \| M5\| M6 \| M7 \| M8	Chapters 1,2,3,4 from Neural Networks by Rojas
Lecture 3	Sigmoid Neurons, Gradient Descent, Feedforward Neural Networks, Representation Power of Feedforward Neural Networks	Slides	M1 \| M2\| M3 \| M4 \| M5\| M6 \| M7\| M8	http://neuralnetworksanddeeplearning.com/chap4.html
Lecture 4	Feedforward Neural Networks, Backpropagation	Slides	M1 \| M2 \| M3 \| M4 \| M5 \| M6 \| M7 \| M8	See Lecture 2 [Training Neural Networks] by Hugo Larochelle
Lecture 5	Gradient Descent (GD), Momentum Based GD, Nesterov Accelerated GD, Stochastic GD, AdaGrad, RMSProp, Adam	Slides-1 \| Slides-2	M1 \| M2 \| M3 \| M4 \| M5 \| M6 \| M7 \| M8 \| M9 \| M10 \| M11 \| M12 \| M13 \| M14\| M15	http://ruder.io/optimizing-gradient-descent/ http://cs231n.github.io/neural-networks-3/ AdaGrad RMSProp
Lecture 6	Bias Variance Tradeoff, L2 regularization, Early stopping, Dataset augmentation, Parameter sharing and tying, Injecting noise at input, Ensemble methods, Dropout	Slides	M1\| M2 \| M3 \| M4\| M5 \| M6 \| M7 \| M8 \| M9 \| M10\| M11\|M12	Chapter 7 from Deep Learning book Dropout Ali Ghodsi's Video Lectures on Regularization
Lecture 7	Greedy Layerwise Pre-training, Better activation functions, Better weight initialization methods, Batch Normalization	Slides	M1 \| M2\| M3 \| M4 \| M5\| M6	Batch Normalization Greedy Layerwise Pre-training Reducing the Dimensionality of Data with Networks Exploring Strategies for Training Deep Neural Networks The difficulty of training deep architectures and effect of unsupervised pre-training Why does Unsupervised Pre-Training Help?
Lecture 8	Convolutional Neural Networks, LeNet, AlexNet, ZF-Net, VGGNet, GoogLeNet, ResNet	Slides	M1 \| M2\| M3 \| M4 \| M5\| M6 \| M7\| M8	Chapter 9 AlexNet ZFNet VGGNet 9 Deep Learning papers GooLeNet ResNet
Lecture 9	Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks	Slides	M1 \| M2\| M3 \| M4 \| M5\| M6 \| M7 \| M8 \| M9 \| M10	RCNN(See the section on visualizations) ZFNet(see the section on experiments) Striving for simplicity(Guided Backpropagation) DeepVis Deep Inside Convolutional Networks Understanding Convolutional Networks Understanding Deep Image Representations DeepDream A Neural Algorithm for Artistic Style Neural style Intriguing properties of neural networks Deep Neural Networks are easily fooled
Lecture 10	Learning Vectorial Representations Of Words	Slides	M1 \| M2\| M3 \| M4 \| M5\| M6 \| M7 \| M8 \| M9 \| M8 \| M9 \| M10\| M11	Deriving Mikiolov's word2vec word2vec parameter learning explained Blog 1 Blog 2 Blog 3
Lecture 11	Recurrent Neural Networks, Backpropagation Through Time (BPTT), Vanishing and Exploding Gradients, Truncated BPTT	Slides	M1 \| M2\| M3 \| M4 \| M5	On the difficulty of training RNNs
Lecture 12	Gated Recurrent Units (GRUs), Long Short Term Memory (LSTM) Cells, Solving the vanidhing gradient problem with LSTMs	Slides	M1 \| M2\| M3 \| M4	LSTM: A Search Space Odyssey Understanding Deriving and Extending the LSTM Understanding LSTM Networks (Nice blog) Paper which introduced Gated Recurrent Units Paper which introduced LSTMs
Lecture 13	Encoder Decoder Models, Attention Mechanism, Attention over images, Hierarchical Attention	Slides	M1 \| M2\| M3	Attention-based Encoder–Decoder Networks
Lecture 14	Transformers: Multi-headed Self Attention, Cross Attention	Slides	M1 \| M2\| M3 \| M4 \| M5

Quizzes/Assigments

	Topics	Resources	Release Date	Submission Date
Assignment 1 (Graded)	Feedforward Neural Networks	Link	19-Feb-2024	10-Mar-2024
Assignment 2 ( Ungraded )	Convolutional Neural Networks	Link	03-Mar-2024	03-Apr-2024
Assignment 3 ( Ungraded )	Recurrent Neural Networks	Link	03-Mar-2024	03-Apr-2024
Assignment 4 ( Ungraded )	RBMs and GANs	Link	03-Mar-2024	03-Mar-2024
Assignment 5 ( Graded )	Transformers		22-Mar-2024	22-Apr-2024
Endsem			11-May-2024 (Saturday)	--

Tutorials

	Topics	Resources
Tutorial 1	Calculus	pdf
Tutorial 2	Linear Algebra	pdf
Tutorial 3	MP Neurons, Perceptrons	pdf
Tutorial 4	Sigmoid Neurons, Gradient Descent	pdf
Tutorial 5	Feedforward Neural Networks, Backpropagation	pdf

What Next?

You may find the following courses to be useful:

Deep Learning for Computer Vision [from Stanford]
Deep Learning for NLP [from Stanford]

CS6910: Deep Learning