Lectures

You can download the lectures here. Lecture slides are likely to be updated before the corresponding class so check back again right before class.

• 01 - Deep Learning Concepts and Course Logistics
  tl;dr: We will introduce the topic of deep learning, a bit about it's history, and what impact it has had. Then we'll go over the course logistics, the lecture topics, homework assignments and projects.
  [slides] [lecture recording]
  

  Readings:

  • Understanding Deep Learning, Chapter 1
• 02 - Supervised Learning
  tl;dr: We go a little deeper into supervised learning, introducing terminology and illustrating with a simple example of a linear model.
  [slides] [lecture recording]
  

  Readings:

  • Understanding Deep Learning, Chapter 2
• 03 - Shallow Networks
  tl;dr: In this lecture we consider networks with one layer of hidden units and explore their representational power.
  [slides] [lecture recording]
  

  Readings:

  • Understanding Deep Learning, Chapter 3
• 04 - Deep Networks
  tl;dr: We dive into deep networks by composing two shallow networks and visualizing their representational capabilities. We then generalize fully connected networks with two and more layers of hidden units. We'll compare the modeling efficiency between deep and shallow networks.
  [slides] [lecture recording]
  

  Readings:

  • Understanding Deep Learning, Chapter 4
• 05 - Loss Functions, Part 1
  tl;dr: In this lecture, we will motivate and derive common loss functions from a probabilistic view point using maximum likelihood estimation.
  [slides] [lecture recording]
  

  Readings:

  • Understanding Deep Learning, Chapter 5

  Extra Readings:

  • Mathematics for Machine Learning, Chapters 6 and 8, especially 8.1.3
  • Maximum Likelihood Estimation Examples from an MIT course. This video walks through MLE examples for binomial and normal distributions, but does not pull out equivalent loss functions like we did in lecture.
• 06 - Loss Functions, Part 2, SCC Tutorial
  tl;dr: In this lecture, we continue to look at loss functions and introduce the shared compute cluster (SCC) for training models.
  [slides] [SCC Overview] [lecture recording]
  

  Readings:

  • Understanding Deep Learning, Chapter 5
• 07 - Gradient Descent and Fitting Models
  tl;dr: In this lecture we look at different ways minimizing the loss function for models given a training dataset. We'll formally define gradient descent, then show the advantages of stochastic gradient descent and then finally see how momentum and normalized gradients (ADAM) can improve model training farther.
  [slides] [lecture recording]
  

  Readings:

  • Understanding Deep Learning, Chapter 6
• 08 - Backpropagation
  tl;dr: In this lecture we show how to efficienctly calculate gradients over more complex functions like deep neural networks using backpropagation.
  [slides] [lecture recording]
  

  Readings:

  • Understanding Deep Learning, Chapters 7.1 - 7.4
• 09 - Initialization
  tl;dr: In this lecture we show how to use gradients to derive better initialization schemes.
  [slides] [how to read a research paper] [pytorch example notebook] [pytorch example notebook colab]
  

  Readings:

  • Understanding Deep Learning, Chapter 7.5
• 10 - Measuring Performance
  tl;dr: We look at measuring model training performance, the importance of test sets as well as how noise, bias and variance play a role in training outcomes.
  [slides]
  

  Readings:

  • Understanding Deep Learning, Chapter 8
  • Double Descent Demystified
• 11 - Regularization
  tl;dr: We explain explicit and implicit regularization techniques and how they help generalize models.
  [slides]
  

  Readings:

  • Understanding Deep Learning, Chapter 9
  • Double Descent Demystified
  • For Valid Generalization the Size of the Weights is More Important than the Size of the Network
  • Train faster, generalize better: Stability of stochastic gradient descent
• 12 - Convolutional Neural Networks
  tl;dr: We cover 1D and 2D convolutional neural networks along with subsampling and upsampling operations.
  [slides]
  

  Readings:

  • Understanding Deep Learning, Chapter 10
• 13 - Residual Networks
  tl;dr: In this lecture we introduce residual networks, the types of problems they solve, why we need batch normalization and then review some example residual network architectures.
  [slides]
  

  Readings:

  • Understanding Deep Learning, Chapter 11
• 14 - Recurrent Neural Networks
  tl;dr: In this lecture we introduce recurrent neural networks, starting the plain vanilla RNN, the problem of vanishing gradients, LSTM and GRU and batch normalization.
  [slides]
  

  Readings:

  • lecture slides
• 16 - Attention and Transformers
  tl;dr: In this lecture we cover the attention mechanism used to handle longer variable length contexts, and present the transformer architure, self-attention and the variations for encoder, decoder and encoder-decoder type models.
  [slides]
  

  Readings:

  • Understanding Deep Learning, Chapter 12
  • Optional: Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)
  • Optional: The Illustrated Transformer
• 17 - Transformers Part 2
  tl;dr: In this lecture we continue to review the transformer architecture. We continue the discussion of decoders and encoder-decoder architectures, then discuss scaling to large contexts and then tokenization and embedding.
  [slides]
  

  Readings:

  • Understanding Deep Learning, Chapter 12
  • Optional The Illustrated Transformer
• 18 - Training, Tuning and Evaluating LLMs
  tl;dr: In this lecture go through the entire LLM training process, starting with pretraining, then finetuning. We'll also discuss how to evaluate LLMs as well as a parameter efficient fine tuning technique called LORA.
  [slides]
  

  Readings:

  • See slides for references
• 19 - Vision & Multimodal Transformers
  tl;dr: In this lecture we'll cover vision and multimodal transformers as a survey of three papers.
  [slides]
  

  Readings:

  • See slides for references
• 20 - Adversarial Inputs and Generative Adversarial Models
  tl;dr: In this lecture we introduce adversarial inputs. We will then dive into Generative Adversarial Networks (GANs) and their applications. We will also discuss the challenges and limitations of GANs and some of the recent advances in the field.
  [slides]
  

  Readings:

  • Intriguing Properties of Neural Networks
  • Robustness and Generalization via Generative Adversarial Training
  • Adversarial Examples are Not Bugs, They are Features
  • Generative Adversarial Nets
  • A Style-Based Generator Architecture for Generative Adversarial Networks
  • Understanding Deep Learning, Chapters 15
• 21 - Unsupervised Learning and Variational Autoencoders
  tl;dr: In this lecture we revisit the concept of unsupervised learning in the context of generative models and then dive into Variational Autoencoders. After reviewing unsupervised learning, we look look at autoencoders and their ability to reduce dimensions of inputs into a latent space. We'll see why they don't make good generative models and then generalize to VAEs. We'll finish with some examples of generative output of VAEs.
  [slides]
  

  Readings:

  • Understanding Variational Autoencoders
  • Understanding Deep Learning, Chapter 14, 17 (optional)
• 22 - Diffusion Models
  tl;dr: In this lecture, we consider diffusion models, the current best practice for image generation.
  [slides]
  

  Readings:

  • Rocca, Understanding Diffusion Probabilistic Models
  • Understanding Deep Learning, Chapter 18
• 23 - Latent Diffusion Models
  tl;dr: In this lecture, we consider diffusion models, the current best practice for image generation.
  [slides]
  

  Readings:

  • Rocca, Understanding Diffusion Probabilistic Models
  • High-Resolution Image Synthesis with Latent Diffusion Models
  • Understanding Deep Learning, Chapter 18
• 24 - Using Pre-Trained Models
  tl;dr: In this lecture we explore the usage of pre-trained models as a foundation for classification and controlled generation.
  [slides]
  

  Readings:

  • LoRA: Low-Rank Adaptation of Large Language Models
  • Adding Conditional Control to Text-to-Image Diffusion Models Examples:
  • ControlNet example notebook
• 25 - Data Preparation and Augmentation
  tl;dr: In this lecture we investigate the use of different strategies preparing data to facilitate learning and generalization.
  [slides]
  

  Readings:

  • Understanding deep learning (still) requires rethinking generalization (access through BU)
  • Leveraging Random Label Memorization for Unsupervised Pre-Training
  • What Do Neural Networks Learn When Trained With Random Labels?
• 26 - Reasoning and World Models
  tl;dr: In this lecture we examine claims that models have world models and can reason, and attempts to encourage such behavior.
  [slides]
  

  Readings:

  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Large Language Models are Zero-Shot Reasoners
  • Learning to Reason with LLMs
  • OpenAI Harmony Response Format
  • DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  • The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
  • DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
  • AlphaGeometry: An Olympiad-level AI system for geometry
  • How Does A Blind Model See The Earth?
  • Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
  • V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

  Extra Readings:

  • s1: Simple test-time scaling
  • Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
  • Language Models, World Models, and Human Model-Building
  • LLMs and World Models
• 27 - Graph Neural Networks
  tl;dr: In this lecture we introduce graph neural networks, define matrix representations, how to do graph level classification and regression, and how to define graph convolutional network layers.
  [slides]
  

  Readings:

  • Understanding Deep Learning, Chapter 13