Lecture 1 · Introduction: What Is a (Deep) Generative Model

Maps to: MIT 6.S978 Deep Generative Models (Fall 2024, MIT EECS; taught by Kaiming He, TA Minghao Guo), Lecture 1 · Thu 09/05/2024 · Introduction. Reading: Lecture 1 slides (lec1_intro.pdf, course page https://mit-6s978.github.io/). The works cited on the slides are listed in §12. Basis: This lecture follows the Lecture 1 slides:

1. examples from the GenAI era and its prehistory (Efros–Leung texture synthesis, PatchMatch);
2. what generative problems have in common;
3. discriminative vs generative models and Bayes’ rule;
4. where probability comes from (latents and world models, probability as part of the modeling, underfitting and overfitting);
5. generative models with probabilistic modeling;
6. the five components of deep generative models;
7. writing real problems as $p(x \mid y)$.

The slides give only these points and figures. These notes add the mathematics behind them — the optimum of squared error, the two directions of Bayes’ rule, KL and maximum likelihood, forward vs reverse KL, conditional entropy — and all the experiments. GlyphWorld and every toy experiment were designed for these notes and do not reflect what was said in class. Code: code/lecture01/ (7 scripts + 3 modules; numpy + PyTorch (CPU) + matplotlib; 1–10 seconds each) Reading time: 4–5 hours including running the code

0. How to read this lecture

0.1 Learning goals

After this lecture you should be able to:

1. Name what problems “needing a generative model” have in common (one-to-many, some answers more plausible than others, no exact answer in the training data, output more complex than input), and explain why regression (minimizing squared error) on such problems yields “the average of all answers” — an answer that often looks like none of them.
2. Distinguish discriminative models $p(y \mid x)$ from generative models $p(x \mid y)$, move between them with Bayes’ rule, and say exactly “what a discriminative model is missing to become a generative one”.
3. Explain where the probability of data comes from with “latents + a world model”: why simple latents passed through a deterministic map produce a complex data distribution, and why fitting simple models directly in pixel space fails.
4. Understand that “probability is part of the modeling”: we only have finitely many samples and the model must extrapolate; recognize underfitting and overfitting in generative models, and know that “a δ function on every training point” means memorizing the training set.
5. Derive “maximizing likelihood = minimizing the KL divergence”, and understand the two different errors (“blur” vs “miss”) that forward and reverse KL produce when the model family is too small.
6. Name the five components of a deep generative model (formulation, representation, objective, optimization, inference), and point out which parts are “designed” and which are “learned” in autoregressive and diffusion models.
7. Write chat, text-to-image, class-conditional generation, classification and robot policies as $p(x \mid y)$, and use conditional entropy to understand “the more abstract $y$, the more generative the problem”.

0.2 Prerequisites and scope

The original course requires a deep-learning course (MIT 6.S898 / 6.7960) plus either computer vision or NLP. This lecture is the map of the whole course and needs very little mathematics:

0.3 Map

§1  the "GenAI era" and its prehistory: Efros-Leung texture synthesis, PatchMatch
§2  which problems need generative models: one-to-many; regression averages; "ghosts"        ← code 01
§3  discriminative vs generative: the two directions of Bayes' rule; "the most-8-like image"  ← code 02
§4  where probability comes from: latents and world models; finite data; under/overfitting   ← code 03, 04
§5  generative models with probabilistic modeling: KL and maximum likelihood; forward vs reverse KL ← code 05
§6  deep generative models: representing distributions; designed vs learned; five components ← code 06
§7  writing real problems as p(x|y): cases and conditional entropy                              ← code 07
§8  the big picture: what this course covers
§9  summary   §10 common pitfalls   §11 exercises   §12 further reading   §13 next lecture

Suggested order: §1 → §2 → run 01 → §3 → run 02 → §4 → run 03, 04 → §5 → run 05 → §6 → run 06 → §7 → run 07 → §8.

0.4 Notation (used throughout the course)

0.5 About 6.S978

6.S978 is a graduate seminar taught by Kaiming He in MIT EECS (“S” marks a special-topics course; these notes follow the Fall 2024 edition). He was previously at Facebook AI Research. He is best known for ResNet, Faster / Mask R-CNN, MoCo and MAE — all “discriminative” or “representation-learning” work. In recent years his research has turned to generative models; for example, MAR, which appears as an example on the slides, is covered in Lecture 8. The course description focuses on computer vision (images, video, geometry), with related fields such as robotics, biology and materials.

The format of the course determines how these notes are written:

• 5 lectures: introduction (this one), VAE, autoregressive models, GANs, energy-based models / score matching / diffusion. Kaiming He teaches the core modeling idea of each family, not the latest tricks.
• 17 paper-reading sessions: 3 papers each, with 20-minute student presentations. The papers range from 1999 texture synthesis to Movie Gen and AlphaFold 3 in 2024.
• 2 guest lectures: Jun-Yan Zhu (data ownership) and Yang Song (consistency models).
• 6 problem sets: VAE, PixelCNN, GAN, DDPM, conditional flow matching, consistency models — all on MNIST.

The last slide says the course covers three things:

1. how real-world problems are formulated as generative models;
2. probabilistic foundations and learning algorithms;
3. challenges, opportunities and open problems.

Note the first one: “how to formulate a problem as a generative model” (formulation) comes first. This perspective runs through the whole course. The same problem can be modeled in many ways, and which one you pick often matters more than which network architecture you use.

0.6 The toy worlds

Real image distributions have no formula, so the code of these notes uses two kinds of data “made by code, with the truth fully known”:

(1) The seven-segment digit world (GlyphWorld, toy_worlds.py). Every image has 16×16 = 256 pixels with gray levels in $[0,1]$. An image is generated in two steps:

1. Draw 7 latents (“knobs”): the digit class (uniform over 0–9), horizontal shift dx, vertical shift dy, size scale, slant, stroke thickness thick, ink intensity ink — each continuous knob uniform on an interval.
2. Hand them to a deterministic renderer of a few dozen lines: light up segments by the seven-segment rules; the closer a pixel is to the nearest lit segment, the brighter it is (with a 1-pixel soft edge).

      a          0: abcdef   1: bc     2: abdeg   3: abcdg   4: bcfg
    ─────        5: acdfg    6: acdefg 7: abc     8: abcdefg 9: abcdfg
 f │     │ b
   │  g  │
    ─────
 e │     │ c
   │     │
    ─────
      d

This is a concrete instance of the “latents → world model → observation” of §4. Because the generating process is known, many quantities that are usually impossible to compute become exact:

• how many possibilities the other half of an image has, given one half;
• whether a generated image is “valid” (read whether each segment is lit, and see whether they spell a digit);
• conditional entropies under various conditions.

The code of the whole course uses it in place of MNIST.

(2) Known 1-D / 2-D distributions: Gaussian mixtures (with a density formula) and a 2-D “sine-wave band”. They demonstrate concepts that need exact densities, such as density estimation and the KL divergence.

1. The “GenAI” era and before

1.1 What today’s generative models do

The slides open with six examples:

These applications span text, images, video, code, molecules and weather, and seem to have nothing in common. §2 explains why they all need “generative models” rather than the familiar “classification / regression models”.

1.2 Prehistory I: texture synthesis in 1999 (Efros–Leung)

Generative models did not appear in the 2020s. The slides deliberately show two “pre-GenAI” examples. The first is the texture synthesis algorithm of Efros and Leung (1999) (Texture Synthesis by Non-parametric Sampling, ICCV 1999).

The problem: given a small piece of texture (a brick wall, a piece of fabric), generate a large new image that “looks like the same texture but is not a simple copy-paste”.

Their method is strikingly simple:

1. Start from a small seed (a piece cut from the example) and grow outward one pixel at a time.
2. To fill a new pixel, look at its already filled neighbours, forming a “small window with holes”.
3. Find all locations in the example texture whose windows are “similar enough” (within a factor $(1+\epsilon)$ of the best match’s distance).
4. Pick one of these locations at random and copy its center pixel.

The random pick in step 4 is crucial:

• always picking the best match makes the texture fall into an endless loop repeating the same fragment;
• averaging the center pixels of all matches makes the texture blurrier and blurrier.

Picking one at random is sampling from “the conditional distribution of the center pixel given its neighbours”. That conditional distribution has no formula; it is represented “non-parametrically” by the matching locations in the example.

The slides’ verdict: “in today’s language, this is an autoregressive model.” Indeed:

• it generates one element after another in some order (“auto”: it uses what it has generated as input);
• each step samples from a conditional distribution $p(\text{new pixel} \mid \text{generated neighbours})$;
• only the conditional distribution is not given by a neural network but estimated by “finding neighbours in the example”.

Twenty-odd years later, GPT generating text and the early PixelCNN generating images use exactly the same skeleton, with “finding neighbours in the example” replaced by “predicting the conditional distribution with a neural network”. Part 2 of script 01 implements an Efros–Leung-style “nearest-neighbour sampling”, and you will see the difference between “sampling” and “averaging” with your own eyes.

1.3 Prehistory II: PatchMatch in 2009 (content-aware fill)

The second example is PatchMatch (Barnes et al., SIGGRAPH 2009), which became the core algorithm behind Photoshop’s “Content-Aware Fill”: circle a person or a telephone pole in a photo, and the software fills the hole with plausible surrounding content.

PatchMatch itself is a fast randomized algorithm for approximate nearest neighbours: for every small patch of an image, find the most similar patch elsewhere. Brute force is too slow. PatchMatch’s insight is that natural images are highly coherent: if a patch has found a good match somewhere, its right-hand neighbour can probably find a good match just to the right of that location. So the algorithm alternates two steps:

• propagation: try “borrowing” the good matches found by neighbours;
• random search: try a few random locations around the current match, with exponentially shrinking radius.

A few iterations converge to very good results.

Like Efros–Leung, PatchMatch’s “generative” ability comes from finding suitable fragments in existing data and stitching them together. It has no explicit probability model, yet it solves an essentially generative problem: “what should be in this hole?” — the answer is not unique; it only has to be plausible.

1.4 Why start from “old methods”

These two examples hint at an undercurrent of the course: many core ideas of today’s deep generative models appeared before deep learning:

• sampling element by element from conditional distributions (autoregression);
• defining distributions by the data themselves (non-parametric methods);
• outputs that are “plausible” rather than “correct”.

What deep learning brings is representational power. Conditional distributions that used to be represented by “finding neighbours in the example” are now represented by neural networks, which generalize to combinations never seen in the training data. The Lecture 2 reading session is devoted to the “image priors” of the pre-deep-learning era.

2. What problems need generative models

2.1 Four common features

The slides ask: what do chatbots, image generation, video generation and protein generation have in common? Four things:

1. One input corresponds to multiple, even infinitely many, predictions. “Draw a cat” can be drawn as countless different cats; “hello” has countless appropriate replies.
2. Some predictions are more “plausible” than others. A five-legged cat is a “possible” output but far less plausible than a normal cat.
3. The training data may contain no exact answer at all. The training set has no image of “a teddy bear wearing an astronaut helmet riding a horse”; the model must compose something new.
4. The prediction is often more complex, more informative and higher-dimensional than the input. A prompt of a dozen words must produce an image with a million pixels.

Compare the familiar discriminative task of image classification: the input is an image, the output one label, “dog”. There is a single “desired output”, and the output (1 of 10 or 1 of 1000) is far simpler than the input (hundreds of thousands of pixels). The discriminative mindset is “for every input, find the correct answer”. A generative task asks instead “for every input, give a distribution and sample from it”.

2.2 A minimal example: given $y = x^2$, guess $x$

Start with the simplest case (part 1 of script 01). $x$ is standard normal; we only see $y = x^2$ and must guess $x$. Every $y > 0$ obviously has two equally plausible answers, $+\sqrt{y}$ and $-\sqrt{y}$.

What does the discriminative habit — train a regression model minimizing the squared error $\mathbb{E}[(x - f(y))^2]$ — give? The experiment (20,000 training samples, binned by $y$):

The regression answers are almost all 0 — the average of the two answers. But $0^2 = 0 \ne y$: for every $y > 0$, the regression answer is an “impossible” $x$. On 1500 test points we measure “plausible” (the absolute value of the prediction within 0.1 of $\sqrt{y}$):

• regression is plausible 0.0% of the time;
• sampling is plausible 99.7% of the time, and gives a positive sign 47.3% of the time — about half each.

2.3 Why regression must “average”

This is not a badly trained regression model; it is dictated by the squared-error objective itself. Let $X$ be the quantity to predict and $Y$ the input. For any predictor $f$:

After expanding the square, the cross term $2\,\mathbb{E}\big[(X - \mathbb{E}[X \mid Y])^\top(\mathbb{E}[X \mid Y] - f(Y))\big]$ is 0: first take the expectation over $X$ given $Y$, and the conditional expectation of the first bracket is 0. The first term does not depend on $f$, and the second is minimized (to 0) by $f(Y) = \mathbb{E}[X \mid Y]$. Therefore:

The squared-error-optimal prediction is the conditional expectation $\mathbb{E}[X \mid Y]$ — the probability-weighted average of all plausible answers.

When the conditional distribution $p(x \mid y)$ has a single sharp peak, the average sits on the peak and regression works well. When $p(x \mid y)$ has several peaks, the average falls between them — somewhere that looks like none of them. This is the direct consequence of feature 1 of generative problems (one-to-many).

2.4 Experiment: completing glyphs, and “ghosts”

Now the same phenomenon in images (part 2 of script 01). Given only the top half of a glyph (rows 0–7), complete the bottom half. The key: 2 and 3, 5 and 6, 8 and 9 have identical top halves — one input, two plausible answers. Five completion methods are compared:

• (A) Oracle regression $\mathbb{E}[\text{bottom} \mid \text{top}]$: use knowledge of the true world to compute the bottom halves of the two candidate digits and average them. Squared-error optimal; unavailable in reality.
• (B) MLP regression: an ordinary neural network, input the top half, output the bottom half, trained with squared error on 20,000 training images.
• (C) Nearest-neighbour average: find the 5 training images with the most similar top halves and average their bottom halves.
• (D) Nearest-neighbour sampling: the same 5 neighbours, pick one at random and copy its bottom half — the Efros–Leung idea.
• (E) Oracle sampling: pick one of the two candidates with probability 1/2 each and draw it in the true style.

Three rulers grade them:

• per-pixel squared error;
• crisp: the three bottom segments c, d, e are each either lit or dark, without half-transparent ghosts;
• valid: top and bottom together read as some digit 0–9.

On the 622 test images with ambiguous top halves:

The figure shows it most clearly. When the original is an 8, regression draws a half-transparent “ghost” where the lower-left segment e should be: the average of an 8 (e lit) and a 9 (e dark). The MLP learns almost exactly the oracle regression (squared error 0.0088 vs 0.0073): the neural network is not wrong; it faithfully learned what you asked it to learn — the conditional expectation. Sampling completes a crisp 8 or 9. About half the time it “guesses wrong” about the original digit, but that “wrong” answer is just as plausible: the top half alone cannot distinguish 8 from 9.

On the 378 unambiguous test images (0, 1, 4, 7), MLP regression is crisp and valid 98.9% of the time: when an input has only one answer, regression is enough and no generative model is needed.

In the real world this “ghost” phenomenon is called blurriness. Image prediction, super-resolution and video prediction trained with an L2 loss are always blurry, precisely because of the averaging seen here. Later lectures connect back to it: VAE samples are blurry (Lecture 3), and GANs introduce an adversarial loss to “de-blur” (Lecture 9).

2.5 Why per-pixel error is a bad ruler

The table holds another intriguing number. Oracle sampling has a higher squared error (0.0152) than oracle regression (0.0073), with a ratio of 2.09. The “about 2×” is no coincidence. Let $X$ be the true bottom half, $S$ an independent draw from the same conditional distribution, and $m = \mathbb{E}[X \mid y]$:

(The cross term vanishes because $S$ and $X$ are independent with mean $m$.) The right-hand $\mathbb{E}\Vert X - m\Vert^2$ is exactly the squared error of oracle regression. So:

A perfect generative model has exactly twice the per-pixel squared error of a perfect regression model.

Ranking generative models by squared error (or the equivalent PSNR) systematically favours “blurry” answers. That is why generative models need other evaluations:

• likelihood (§5);
• distances between the sample distribution and the real distribution (FID-type metrics, prerequisite P2);
• human preference studies.

2.6 Copying nearest neighbours: how much data is enough?

Nearest-neighbour sampling (D) is only 85.5% crisp on ambiguous inputs, not 100%. Why? Because it can only copy. No training image has exactly the style (position, thickness, slant, …) of the test image, so the copied bottom half does not line up with the visible top half, and the segment reader lands on “half-lit” positions. Part 3 of script 01 varies the training-set size:

More data means a more similar nearest image and better results, but the growth is slow: a 30× larger training set only halves the nearest-neighbour distance. Here there are only 6 continuous style factors. Real images have far more “style” dimensions, and copying would need impossibly much data.

This is what feature 3 of generative problems (no exact answer in the training data) means: the model must interpolate between training samples and compose new things. Non-parametric copying cannot do that, so we need parametric models that generalize — neural networks.

But Efros–Leung’s idea still stands: for one-to-many problems, draw one from the conditional distribution instead of averaging.

2.7 Summary

The essence of a generative problem: for one input, there is a distribution of plausible answers. Regression (minimizing squared error) gives the mean of that distribution, which looks like nothing when there are several peaks; a generative model must represent the distribution and sample from it. Per-pixel error favours the mean and cannot evaluate generation.

3. Discriminative vs. generative models

3.1 Two directions

The slides draw the two kinds of models as arrows in two directions:

• Discriminative model: input a “sample” $x$ (an image), output a “label” $y$ (“dog”). It learns $p(y \mid x)$; for every input, one desired output.
• Generative model: input a “label” $y$ (“dog”), output a “sample” $x$ (an image of a dog). It learns $p(x \mid y)$; for every input, many possible outputs.

The meanings of $x$ and $y$ do not change: $x$ is always “data / sample”, $y$ always “label / condition”. What changes is the direction of the arrow.

3.2 A generative model can be used as a discriminative one: Bayes’ rule

Given a generative model $p(x \mid y)$ and the prior $p(y)$ (how common each class is), Bayes’ rule immediately gives a discriminative model:

The slides point out that the denominator $p(x)$ “is a constant for a given $x$”: to classify, we only compare $p(x \mid y)p(y)$ across $y$, without computing $p(x)$. So “a generative model can be discriminative” (given the prior).

3.3 Can a discriminative model be used as a generative one? It needs $p(x)$

In the other direction, with a discriminative model $p(y \mid x)$ we want $p(x \mid y)$:

This time the denominator $p(y)$ is a constant for a given $y$, no problem. The trouble is the extra $p(x)$ in the numerator — the distribution of the data itself, which a discriminative model never models. The slides conclude: “for a discriminative model to become generative, one still needs to model the prior distribution of $x$.” And $p(x)$ (“the distribution of all natural images”) is exactly the hardest thing to model. Hence:

The challenge is about representing and predicting distributions.

Neither direction escapes “representing a distribution”: a generative model must represent $p(x \mid y)$, and a discriminative model that wants to generate must supply $p(x)$.

3.4 Experiment: one dataset, two classifiers

Script 02 compares two classifiers in a 2-D two-class world (each class a Gaussian; during training $p(y=1) = 0.3$):

• Generative classifier: fit a Gaussian $p(x \mid y)$ per class (sample mean, sample covariance) plus the class frequencies $p(y)$, and classify with Bayes’ rule (in statistics, quadratic discriminant analysis, QDA).
• Discriminative classifier: logistic regression fitting $p(y \mid x)$ directly, with quadratic features so that both decision boundaries belong to the same family.

The two routes almost tie. The generative one has a slightly lower log loss here because its assumption (each class is Gaussian) happens to be true. On real data the discriminative one usually wins: “getting the distribution of $x$ right” is much harder than “only getting the boundary right”, and a discriminative model only needs the latter. This also explains why the first decade of deep learning was dominated by discriminative models.

What if the prior changes? At test time the class proportion changes from 0.3 to 0.8:

The generative classifier only needs to swap $p(y)$ to almost match the Bayes optimum, because it models “what $x$ looks like within each class” and “how common each class is” separately. The discriminative one can be corrected too, but the correction $p'(y \mid x) \propto p(y \mid x)\, p'(y)/p(y)$ itself is derived from Bayes’ rule and quietly uses the generative view that ”$p(x \mid y)$ does not change”.

3.5 Experiment: “generating” with a discriminative model

Next we try $p(x \mid y=1) \propto p(y=1 \mid x)\,p(x)$ (part 3 of script 02), comparing three sources:

(a) shows that the formula is right: given $p(x)$ (here a large pool of unlabeled samples), reweighting yields the correct conditional distribution. (b) is what happens “without $p(x)$”. A discriminative model only knows “which side of the decision boundary is class 1”, so the points “most like class 1” spread over the whole half plane, and 90% land where real data never appear.

“The most-8-like image” (part 4 of script 02) applies the same idea to images. Train a digit classifier on GlyphWorld (test accuracy 100%) and ask it “what does the most 8-like image look like to you?” Starting from an all-gray image, run gradient ascent on the input pixels to maximize $\log p(y=8 \mid x)$.

The classifier is 1.0000 confident about all three images, yet they are not digits at all. They lie 0.418, 0.457 and 0.489 from the nearest real image, 6–8 times the distance of real test images (0.064). A discriminative model learns “the boundary between 8 and the other digits”; maximizing $p(y \mid x)$ finds the most extreme point on the far side of the boundary, not a typical 8 — a close relative of “adversarial examples”. Multiply in $p(x)$ (a pool of real images) and sample $\propto p(y \mid x)\,p(x)$, and real 8, 1 and 7 appear at once (bottom row).

3.6 Summary

Generative model + prior → discriminative model (Bayes’ rule); discriminative model + $p(x)$ → generative model. Both directions need to represent a distribution, and the distribution $p(x)$ of high-dimensional data is the hardest part. This is the central problem of the course.

4. Where does probability come from?

4.1 Latent variables and “world models”

We keep saying “the data distribution $p_{\text{data}}$”, but why would a photo have a “probability”? The slides’ first answer: assume the data are produced by some generative process.

• There are latent factors $z$: an object’s pose, lighting, size, …
• $z$ follows a simple distribution (e.g. independent uniform or Gaussian factors).
• The observation $x$ is rendered by a “world model”: $x = g(z)$, where $g$ is a function of $z$ (physics, imaging, a renderer, …).
• So $x$ has a complex distribution.

The slides’ figure comes from Freeman and Tenenbaum’s bilinear models (Learning Bilinear Models for Two-Factor Problems in Vision). The same letter (“content”) written in different fonts (“style”), or the same face photographed under different lighting, can be described as a bilinear combination “style factor × content factor”. It is an early representative of the idea of “decomposing observations into a few simple factors”.

So the slides say: probability is part of the modeling. “The data follow some distribution” is itself a modeling assumption we make. The assumption is useful because it lets us talk about “which data are more common, more plausible”.

4.2 Experiment: simple $z$, complex $x$

GlyphWorld is exactly such a “latents + world model” (script 03). All 7 latents are uniform:

After the renderer, the distribution of the 256 pixels becomes quite complex:

• Single pixels look like no simple distribution. 74.0% of pixel values are exactly 0 (pure background). A particular pixel (row 3, column 8) is 0 28.7% of the time, above 0.6 37.1% of the time, and in between otherwise — a mixture of “a big lump at 0 + a spread of bright values”.

• Pixels have long-range dependencies. Neighbouring pixels have a mean absolute correlation of 0.475, and pixels 12 apart still 0.118. Two concrete examples:

  • two pixels at the ends of the top bar a (5 columns apart) have correlation +0.771: the same stroke lights up together;
  • two pixels on the left and right of the same row (8 columns apart) have −0.499: an overall shift makes the left bright when the right is dark.

  These long-range dependencies come from “one set of latents controlling all pixels at once”.

• The data lie on a curved, low-dimensional “surface”. Principal component analysis (PCA) needs 16 / 27 / 41 / 84 directions to explain 80% / 90% / 95% / 99% of the pixel variance, while these images were generated with only 7 knobs. The set the data occupy (a manifold) is 7-dimensional, but curved, and enclosing it in a flat subspace takes dozens of dimensions. The complexity of “simple → complex” comes from the nonlinearity of the renderer.

4.3 What if we fit simple models directly in pixel space?

The data distribution is complex — can we ignore the latents and fit a simple model directly in pixel space? Part 3 of script 03 tries three. It measures realism by “the per-pixel RMS distance to the most similar of 20,000 real images”: fresh real images give about 0.066 (the “floor”), pure random noise about 0.49.

• (i) Every pixel’s distribution is right, yet together they form random speckle: all dependencies between pixels are lost.
• (ii) One big Gaussian captures “pairwise linear correlations”; its samples are a faint “superposition of all digits”.
• (iii) Knowing the digit, an outline appears, but the strokes are blurry and speckled.
• (iv) Only simple latents plus a nonlinear map produce real digits.

An interpolation experiment shows further what “latent space” means. Take two images $x_A$, $x_B$ of the same digit in two styles:

The average of two plausible images in pixel space is often not a plausible image: two offset digits superimpose into a “double image” (second-to-last row of the figure). Averaging in latent space and then rendering gives a digit “shifted halfway” (last row). In other words, the set of plausible images is not “convex” in pixel space, but “flat” in latent space. This is another face of the ghosts of §2: “averaging” in pixel space is not a good operation.

This leads to what deep generative models must do: learn a map from a simple distribution to a complex one (§6). GlyphWorld’s renderer was written by us. The real world’s “renderer” (physics, optics, human handwriting habits, …) cannot be written by anyone and must be learned from data.

4.4 Perhaps there is no “true distribution” at all

The slides then give a more cautious formulation:

• perhaps there is no “underlying” true distribution at all (what is the distribution of “all cat photos”? It depends on who took them, when, with which camera…);
• even if there is, we only observe finitely many data points;
• the model must extrapolate a distribution from these finite observations;
• so generative models, like discriminative ones, can overfit and underfit.

The point: $p_{\text{data}}$ is a convenient mathematical fiction. What we really have is a dataset $\{x_1, \dots, x_N\}$. “Learning a distribution” is a guess about “what lies behind these points”, and how good the guess is depends on the modeling assumptions.

4.5 Experiment: underfitting, overfitting and δ functions

The slides illustrate this with a few 1-D pictures. For the same few data points, a very wide smooth curve underfits, and a curve with a spike on every point overfits. In the extreme, “a δ function on every data point” equals drawing a random training example — memorizing the training set.

Script 04 turns this picture into an experiment. The true distribution is a mixture of three Gaussians (weights 0.5, 0.3, 0.2; means −2, 1, 3), and only $N = 100$ samples are given. It is estimated with kernel density estimation (KDE): put a small Gaussian with standard deviation $h$ (the bandwidth) on every sample and average:

$h$ is the “how flexible is the model” knob: a large $h$ is smooth (underfitting), a small $h$ spiky (overfitting), and $h \to 0$ puts a δ function on every sample.

For reference:

• the true distribution’s own held-out log-likelihood is −1.655;
• the median distance from fresh real samples to the nearest training point is 0.0159;
• the “copy distance” is the median distance from model samples to the nearest training point.

Four things to notice in this table:

1. The training likelihood rises monotonically as $h$ shrinks and tends to $+\infty$ as $h \to 0$. The reason is simple: at a training point $x_i$ itself, $q_h(x_i) \ge \frac{1}{N}\cdot\frac{1}{\sqrt{2\pi}h} \to \infty$. Judged only by “the likelihood of the training data”, the best model memorizes the training set.
2. The held-out likelihood and the KL first improve, then get worse, with the best value at $h = 0.2$. A too-large $h$ smears the three peaks into one blob. A too-small $h$ leaves large “almost-zero-density” gaps between training points, and the log-likelihood of new data falling there collapses (−99 at $h = 0.005$).
3. An overfitted model copies when it generates. At $h = 0.005$, 97.1% of the generated samples are within 0.01 of a training point (only 37.2% of real new samples are). It can “generate”, but nothing new.
4. Even “a rule for choosing the bandwidth” hides a modeling assumption. The textbook Silverman rule of thumb gives $h = 0.782$, several times larger than the best. The rule is derived assuming “the data look like one Gaussian”, and is wrong for three-peaked data.

One more noteworthy detail: the $h$ chosen by the held-out likelihood and the $h$ chosen by the KL are identical. We cannot compute the KL without the formula of $p_{\text{data}}$, but we can compute the held-out likelihood — and §5 proves that the two differ only by a constant.

4.6 More data, more flexible models

Part 2 of script 04 varies the sample size $N$, each time choosing the best bandwidth on a validation set:

($N = 1000$ and $5000$ land on the same candidate bandwidth because there are only 16 candidates.) The more data, the more flexible the right model and the more accurate the learned distribution. “How flexible is right” is not decided by the model itself but by the amount of data — exactly the bias–variance trade-off of discriminative models. Deep generative models face the same problem, only “flexibility” is no longer a single knob $h$ but the architecture, the number of parameters, the training steps, regularization… Lecture 2’s Deep Image Prior gives a striking example: the network architecture itself is a strong prior.

5. Generative models with probabilistic modeling

5.1 The pipeline

The slides describe the pipeline of “generative models with probabilistic modeling” with a sequence of figures:

  data {x_1, ..., x_N}
        │  ← "the data follow some distribution": this step is itself modeling
        ▼
  the data distribution p_data
        │  ← optimize a loss (measuring the gap between two distributions)
        ▼
  the estimated distribution p_θ
        ├──→ sampling: generate new "data"
        └──→ probability density estimation: p_θ(x) = ? (optional)

The last step has two uses: sampling (generate new samples) and estimating probability densities (given an $x$, how likely is it — for anomaly detection, compression, comparing models). Some models do both (autoregressive models, normalizing flows), some only sample (GANs), some only give a lower bound on the density (VAEs).

The slides add four “notes” here, worth taking one at a time:

1. The statistical models in generative models are often designed and derived by humans. “Data = latents + rendering”, “each conditional is Gaussian”, “the noising process is like this” — all human choices.
2. Probabilistic modeling is not only the neural network’s job. The network is responsible for only one part (§6.3).
3. Probabilistic modeling is a popular approach, but not the only one. For example, a GAN’s generator never writes down $p_\theta(x)$ explicitly, and Efros–Leung has no explicit probability model at all.
4. “All models are wrong, but some are useful.” — the statistician George Box.

5.2 Measuring the gap between two distributions: KL divergence and maximum likelihood

The “loss” in the pipeline must measure how far apart $p_{\text{data}}$ and $p_\theta$ are. The classic choice is the KL divergence (Kullback–Leibler divergence):

It is always $\ge 0$, and 0 only when the two distributions are equal everywhere (provable with Jensen’s inequality, exercise 5). But we do not know the formula of $p_{\text{data}}$ — how can we compute it? Split the log:

The first term is the negative entropy of the data distribution, a constant independent of the model. So:

Minimizing the KL divergence = maximizing the likelihood, and the latter needs only data samples and the model’s density, not the formula of $p_{\text{data}}$. This is the one-line “tl;dr” at the start of the VAE lecture’s slides (original Lecture 2, Lecture 3 of these notes). Equivalently, the mean negative log-likelihood (NLL) satisfies

Part 1 of script 05 verifies this identity numerically. The true distribution is $0.6\,\mathcal{N}(-2, 0.6^2) + 0.4\,\mathcal{N}(2.5, 0.8^2)$. For five different models, the NLL comes from 200,000 samples and the KL from numerical integration:

The last column equals the entropy of the data distribution $H(p_{\text{data}}) = 1.6944$ for every model (differences come only from Monte Carlo error). Two consequences:

• ranking models by NLL is ranking them by KL;
• the floor of the NLL is the entropy of the data, not 0 — the data themselves are uncertain, and no model can remove that.

This also explains why “the held-out likelihood and the KL chose the same bandwidth” in §4.5.

5.3 The two directions of KL: blur or miss

The KL divergence is asymmetric: $D_{\text{KL}}(p \Vert q) \ne D_{\text{KL}}(q \Vert p)$. The slides of the VAE lecture have a side note: “could we consider criteria other than KL?” Before answering it, look at what happens when the direction is switched.

• Forward KL $D_{\text{KL}}(p \Vert q) = \int p \log(p/q)$: where $p > 0$ but $q \approx 0$, $\log(p/q)$ blows up — wherever the data are, the model must be. It punishes “missing data” and does not much care if the model puts probability where there is no data. (Also called “mass-covering”.)
• Reverse KL $D_{\text{KL}}(q \Vert p) = \int q \log(q/p)$: heavily punished where $q > 0$ but $p \approx 0$ — wherever the model is, the data must be. It punishes “inventing things” and does not much care about missing part of the data. (Also called “mode-seeking”.)

When the model family is large enough to contain the true distribution, both directions are optimized by $q = p$, and there is no difference. The difference appears when the model family is too narrow. Part 2 of script 05 fits one Gaussian to two peaks:

(“Samples between peaks”: the fraction of model samples where the true density is < 0.02; “missed data”: the fraction of real data where the model density is < 0.02.)

• The forward-KL solution is “moment matching”: the same mean and variance as the data, one wide Gaussian spanning both peaks. It misses almost no data, but about a third of its samples fall between the peaks — blur. This is the distribution-level version of the “ghosts” of §2: to cover every possibility it has to smear across the middle. Part of the reason why models trained by maximum likelihood (VAEs, autoregressive models) produce “blurry”, “average” samples lies here.
• Reverse KL, started from either side, shrinks onto the nearer peak: its samples are almost all “real” (only 2–4% fall between the peaks), but the other peak’s data are dropped entirely (40% or 60%) — miss. This is the seed of the “mode collapse” of the GAN lecture.
• A result these notes did not anticipate: started right in the middle at $\mu = 0$, reverse KL stops at a “wide, fat” solution (unchanged even after 10,000 steps). A scan over $\mu$ (best $\sigma$ for each $\mu$) shows that reverse KL has three local optima here: two narrow solutions (KL 0.51, 0.91) and this wide one (KL 1.10). The landscape of reverse KL is bumpy, and the result depends on the initialization.

Part 3 of script 05 switches the family to “a mixture of two Gaussians” (which contains the truth). Both directions recover the true parameters: weights 0.61/0.39 and 0.60/0.40, with KL below 0.001 in both cases.

5.4 “All models are wrong”

These experiments make George Box’s remark concrete. In real problems the model family never contains $p_{\text{data}}$ exactly, so the choice of objective decides which kind of error the model makes — blur (forward KL / maximum likelihood) or miss (reverse KL).

There is also a practical difference. Forward KL needs only data samples. Reverse KL must evaluate $\log p_{\text{data}}$ on the model’s samples — and in reality there is no such formula. Later families answer this in different ways:

• GANs (Lecture 9) get around it with a discriminator, and can be shown to optimize a different divergence (the JS divergence);
• diffusion models and flow matching (Lectures 12, 16) give a third kind of answer: they turn learning a distribution into a sequence of regression problems.

6. Deep generative models

6.1 Deep learning is representation learning

The slides place deep learning in one sentence: deep learning is representation learning. It has two kinds of “representation”:

1. Learning to represent data instances: map data to features $x \mapsto f(x)$, then train with a goal-directed loss, e.g. the classification loss $\mathcal{L}(f(x), y)$. This is what discriminative deep learning does: ResNet represents an image as features, followed by a classifier.
2. Learning to represent probability distributions: map a simple distribution (Gaussian / uniform) to a complex one, $z \sim \pi(z)$, $x = g(z)$, so that the distribution pushed forward by $g$ is close to the data distribution. The loss is some gap to the data distribution, $\mathcal{L}(p_\theta, p_{\text{data}})$. This is what generative deep learning does.

And the two are often done together. A VAE’s encoder “represents data instances” (compresses an image into a latent vector) while its decoder represents the distribution. The tokenizers of today’s text-to-image models (Lectures 3, 7, 9) first represent images as compact latents, and the generative model then represents the distribution in that latent space.

The second kind of representation is exactly what §4 described: “simple latents + a nonlinear map = a complex distribution”. In GlyphWorld the map is the renderer we wrote. What a deep generative model must do is learn such a map from data with a neural network — or learn something equivalent: a sequence of conditionals, a denoiser, a velocity field, …

6.2 Not every part is learned

The slides then stress an easily overlooked fact: many parts of distribution modeling are not learned but designed. Two cases:

• Autoregressive models: writing $p(x_1, \dots, x_n)$ as $p(x_1)\,p(x_2 \mid x_1)\,p(x_3 \mid x_1, x_2)\cdots$ — this dependency graph (who depends on whom, in which order) is designed by people; the mapping of each conditional is learned (e.g. by a Transformer).
• Diffusion models: the “noising” process from data to noise and the dependency graph of the “denoising” direction are designed by people; the mapping of each denoising step is learned (e.g. by a U-Net).

This is why the “formulation” mentioned in §0.5 matters: the same network architecture combined with different dependency graphs / noising processes gives entirely different models. Script 06 shows the weight of “design” with a concrete example.

6.3 Five components

Putting the discussion together, the slides list five parts a deep generative model may contain:

Every model in the course can be dissected with this table. Take a GAN:

• formulation = noise mapped to samples by a generator;
• representation = two networks, a generator and a discriminator;
• objective = the adversarial loss;
• optimization = alternating gradient steps of a minimax game;
• inference = sampling with one forward pass, with no density estimate.

6.4 Experiment: a deep generative model that is “small but complete”

Script 06 builds the smallest autoregressive model on a 2-D distribution: $x_1$ uniform, $x_2$ along a sine curve with a little noise. It maps the five parts onto code one by one:

1. Formulation: cut $[0,1)^2$ into 32×32 cells (discretization) and use the chain rule $p(x_1, x_2) = p(x_1)\,p(x_2 \mid x_1)$.
2. Representation: $p(x_1)$ as 32 learnable numbers (logits); $p(x_2 \mid x_1)$ as a small MLP (input the cell index of $x_1$, output logits over the 32 cells of $x_2$).
3. Objective: the negative log-likelihood $-\log p(x_1) - \log p(x_2 \mid x_1)$ — the sum of the cross-entropies of two “classification” problems.
4. Optimization: Adam, minibatches, 2000 steps.
5. Inference: sampling = draw $x_1$, then $x_2 \mid x_1$ (“ancestral sampling”); density = compute $p(x_1)\,p(x_2 \mid x_1)$ directly.

Then five variants, with the same data and the same training:

(Floor: the entropy of the discretized true distribution = 3.7048 bits/dim; two independent uniforms = 5 bits/dim.)

• A and B are both close to the entropy floor: the chain rule holds for any order, and with flexible categorical conditionals both orders learn well.
• C (unimodal Gaussian, $x_1$ first) is also very good, even slightly better than A. Given $x_1$, $x_2$ is a small blob near the sine curve, which one Gaussian fits exactly; and with only 2 numbers per conditional it overfits less than 32 logits.
• D (unimodal Gaussian, $x_2$ first) clearly fails. Given $x_2$, $x_1$ can be at two or three places on the sine wave (multimodal), and one Gaussian can only smear them together (the forward KL of §5.3). Same data, same network, only the factorization order changed. The autoregressive lecture’s slides (original Lecture 3, Lecture 5 of these notes) put it this way: “both factorizations are correct, but one may be much easier to learn than the other”. The dependency graph is designed, and its quality decides success or failure.
• E (no dependency): both marginals are right, but their relationship is lost; samples form a big block instead of a wave — just like “independent pixels” in §4.3.

Finally, the same model both samples and gives the exact probability of any cell. Model A gives 0.00886 for a cell on the band (true value 0.00858), and $2.2\times10^{-7}$ for a cell far from the band (true value 0). An autoregressive model’s “inference” has both.

Why “bits per dimension”? The negative log-likelihood in base 2, divided by the dimension, is “how many bits on average to encode one dimension”. It ties directly to compression: a good probability model is a good compressor. The “bits/dim” common in image-generation papers is this quantity.

7. Formulating real-world problems as $p(x \mid y)$

7.1 The roles of $y$ and $x$

The last part of the slides unifies all these applications in one sentence: a generative model is $p(x \mid y)$.

7.2 Cases

The slides list these cases one by one (grouped slightly here):

A few points worth pausing on:

• “Unconditional” generation has a condition. $p(x)$ is the distribution of “all images”; an “unconditional” model trained on CIFAR-10 learns $p(x \mid y = \text{"like CIFAR-10"})$. The condition hides in the choice of dataset.
• Classification is also $p(x \mid y)$, with the roles swapped: the image is the condition $y$, the class the $x$ to generate. From this angle a discriminative model is a special case of a generative one — with a tiny output space and an almost deterministic conditional.
• Open-vocabulary recognition shows that even “recognition” can have non-unique answers: for a photo of a flamingo, “bird”, “flamingo”, “pink” and “standing in water” are all plausible descriptions. Once the output space becomes open and complex, recognition becomes a generative problem too.
• Robot policies: under the same observation the robot may go around the left or the right; demonstration data are multimodal. Writing the policy as $p(\text{action} \mid \text{observation})$ and representing it with a generative model (Lecture 21) avoids the “averaging” of §2 that would steer the robot into the obstacle.

The slides close with three questions: What is $x$? What is $y$? How do we represent $x$, $y$ and the dependency between them?

7.3 Experiment: the more abstract $y$, the more “generative” the problem

”$y$ is more abstract, with less information” can be measured precisely with information theory. The conditional entropy $H(x \mid y)$ measures “how much uncertainty about $x$ remains once $y$ is given” (in bits; $2^{H}$ can be roughly read as “the average number of equally plausible answers”):

Script 07 builds a small glyph world where every possibility can be enumerated: 10 digits × 3 dx × 3 dy × 2 thicknesses × 3 slants = 540 equally likely combinations, rendering 540 distinct images, so $H(x) = \log_2 540 = 9.077$ bits. For the same $x$ (the image), seven different $y$:

As $y$ goes from “nothing” to “all latents”, $H(x \mid y)$ falls from 9.08 all the way to 0. The more abstract $y$ and the less information it carries, the more possibilities $x$ has and the more “generative” the problem; when $y$ determines $x$ completely, the problem degenerates into a function — and regression is enough (the unambiguous inputs of §2.4). The two “image conditions”, super-resolution and completion, carry a lot of information but still leave a little ambiguity (1.5–2.7 candidates on average, at most 26) — exactly why they need generative models.

Now classification ($x$ = the digit label, $y$ = the image):

Given the full image the label’s entropy is 0: classification is a $p(x \mid y)$ whose “output is almost certain”. Given only the top half, about 0.6 bit remains:

• with the digit centered, it is a choice between two: 2/3, 5/6 or 8/9;
• with the digit shifted down, even the middle bar g is invisible, and it is a choice among three: 0/8/9 or 2/3/7;
• with the digit shifted up, a piece of the bottom half shows, and there is no ambiguity.

A final observation: $H(\text{image}) = 9.077 = H(\text{label}) + H(\text{image} \mid \text{label}) = 3.322 + 5.755$. “Generating an image from a label” is hard precisely because the extra ~5.8 bits must be filled in — what the slides mean by “the output is more complex and more informative than the input”. In real text-to-image, a sentence may carry only tens to hundreds of bits, while an image carries far more.

Script 07 ends by enumerating / sampling directly from several conditional distributions, so you can see that “several $x$ under the same $y$ all satisfy the condition and yet differ from each other” — exactly what $p(x \mid y)$ must represent. Real problems cannot be enumerated (images have thousands of dimensions, styles are continuous), so a neural network must represent the conditional distribution, and we must be able to sample from it.

8. The big picture

8.1 Three things this course covers

The last slide: this course covers

1. how real-world problems are formulated as generative models (§7 of this lecture; the applications of Lectures 19–24);
2. probabilistic foundations and learning algorithms (the model families of Lectures 3–18);
3. challenges, opportunities and open problems (throughout the reading sessions).

8.2 The model families at a glance

Later lectures develop every row of this table. For now, remember that they all answer the question of §6.3: “how do we factorize a complex distribution into simple ones?”

Yang Song’s guest lecture (Lecture 18) has a slide “a catalogue of one-step generative models” that is a subset of this table:

• VAEs: stable training and likelihoods, but low sample quality.
• GANs: high sample quality, but unstable training and no likelihood.
• Flows: exact likelihoods, but restricted architectures and middling sample quality.
• Consistency models: stable training and high sample quality, but no likelihood.

No family dominates across the board, which is why this course compares them side by side.

8.3 This lecture’s place in the course

 Lecture 1 (this one): generation = represent and sample p(x|y); five components
     │
     ├─ Lecture 2: how "priors of natural images" were modeled before deep learning
     ├─ Lecture 3: latents (VAE / EM / VQ-VAE) -- the "latents + world model" of §4, learned by networks
     ├─ Lecture 4: invertible maps (flows) -- the "simple → complex distribution" of §6, with an invertible map
     ├─ Lectures 5-8: the chain rule (autoregressive) -- §6.4 scaled up; tokenizers; combining with diffusion
     ├─ Lectures 9-11: adversarial -- no KL, a network as the loss (the other path of §5.3)
     └─ Lectures 12-18: diffusion and flows -- "simple → complex" split into many small steps

9. Summary

1. The hallmark of a generative problem is one-to-many: one input has a distribution of plausible answers. Regression (squared error) gives the mean of that distribution, which looks like nothing when there are several peaks (“ghosts”); a generative model must represent the distribution and sample from it. Per-pixel error inherently favours the mean: a perfect sampler has exactly twice the squared error of a perfect regressor.
2. Discriminative and generative models are linked by Bayes’ rule: generative model + prior → discriminative model; discriminative model + $p(x)$ → generative model. Gradient ascent on a discriminative model’s input finds the extreme of a boundary, not a typical sample. The challenge is representing and predicting distributions.
3. Probability is part of the modeling.
   • Simple latents passed through a nonlinear “world model” produce a complex data distribution; fitting simple models directly in pixel space fails, and latent space is the “flat” coordinate system.
   • We only have finite data, so the model must extrapolate; hence underfitting and overfitting. δ functions = memorizing the training set.
   • The more data, the more flexible the model can be.
4. Maximum likelihood = minimizing $D_{\text{KL}}(p_{\text{data}} \Vert p_\theta)$; NLL = entropy + KL. With a too-small model family, forward KL makes the model “blur” (cover all data) and reverse KL makes it “miss” (mode collapse). All models are wrong; the objective decides which error they make.
5. Deep generative model = representing a distribution with deep networks: map a simple distribution to a complex one. Five components: formulation, representation, objective, optimization, inference. Dependency graphs, noising processes and the like are often designed; only the map is learned, and the quality of the design can decide success or failure.
6. Everything is $p(x \mid y)$: $y$ is more abstract with less information, $x$ more concrete with more. The larger the conditional entropy $H(x \mid y)$, the more “generative” the problem; at $H(x \mid y) = 0$ it degenerates into a function. Classification is a special case of $p(x \mid y)$.

10. Common pitfalls

1. “Generative model = unsupervised learning.” No. Today’s most important generative models (text-to-image, chat) are conditional generation $p(x \mid y)$; even “unconditional” generation has an implicit condition (§7.2). Whether a generative model is needed depends on whether the output is one-to-many, not on whether there are labels.
2. “The best generation outputs the most probable sample.” No. §2 showed that “averaging” fails; “the most probable point” (the mode) fails too. It has no diversity, and in high dimensions the most probable point is often highly atypical: the mode of a high-dimensional standard Gaussian is the origin, while almost all samples are about $\sqrt{d}$ away. Generation needs sampling.
3. “Lower squared error / higher PSNR is better, so generative models should compete on these.” For generative problems these metrics systematically favour “blurry” answers (§2.5).
4. “The higher the training likelihood, the better the model.” The training likelihood can be pushed to infinity by memorizing the training set (§4.5). Look at the held-out likelihood, or at sample-level evaluations.
5. “A discriminative model’s $p(y \mid x)$ already contains the distribution of the data.” It does not. $p(y \mid x)$ is “insensitive” to $p(x)$: however $p(x)$ changes, as long as the class proportions at each $x$ stay the same, $p(y \mid x)$ does not change. So a discriminative model cannot be used directly to generate (§3.5).
6. “The KL divergence is a distance between two distributions.” It is asymmetric ($D_{\text{KL}}(p\Vert q) \ne D_{\text{KL}}(q \Vert p)$) and violates the triangle inequality, so it is not a distance in the mathematical sense; its two directions have completely different personalities (§5.3).
7. “Everything in a deep generative model is learned.” Dependency graphs, factorization orders, noising processes, discretizations, tokenizers… many key choices are designed by people (§6.2, §6.4).
8. “Autoregressive models are RNNs / only for time series.” Autoregression is a formulation (the chain rule), independent of the network architecture and not requiring a temporal order; the order and the partition can be chosen freely (Lecture 5).
9. “The average of two images in pixel space should also be a plausible image.” The set of plausible images is not convex in pixel space (the interpolation experiment of §4.3). Interpolation is meaningful in latent space — provided the latent space is learned well.

11. Exercises

Exercise 1 (squared error and the conditional expectation) (a) Prove the decomposition of §2.3, and deduce that $\arg\min_f \mathbb{E}\Vert X - f(Y)\Vert^2 = \mathbb{E}[X \mid Y]$. (b) Prove: if $S$ and $X$ are i.i.d. given $Y$, then $\mathbb{E}\Vert S - X\Vert^2 = 2\,\mathbb{E}\Vert X - \mathbb{E}[X \mid Y]\Vert^2$. (c) If the squared error is replaced by the mean absolute error $\mathbb{E}|X - f(Y)|$ (1-D), what is the optimal $f(Y)$? For the $y = x^2$ problem of §2.2, does it still give “impossible answers”?

(a) Let $m(Y) = \mathbb{E}[X \mid Y]$. $\Vert X - f\Vert^2 = \Vert (X - m) + (m - f)\Vert^2 = \Vert X - m\Vert^2 + \Vert m - f\Vert^2 + 2(X-m)^\top(m - f)$. Take the expectation of the cross term given $Y$ first: $m - f$ is a function of $Y$ and can be pulled out, while $\mathbb{E}[X - m \mid Y] = 0$, so the cross term has expectation 0. Hence $\mathbb{E}\Vert X - f\Vert^2 = \mathbb{E}\Vert X - m\Vert^2 + \mathbb{E}\Vert m - f\Vert^2 \ge \mathbb{E}\Vert X - m\Vert^2$, with equality iff $f(Y) = m(Y)$ (almost everywhere).

(b) Given $Y$, $\mathbb{E}\Vert S - X\Vert^2 = \mathbb{E}\Vert (S - m) - (X - m)\Vert^2 = \mathbb{E}\Vert S - m\Vert^2 + \mathbb{E}\Vert X-m\Vert^2 - 2\,\mathbb{E}[(S-m)]^\top \mathbb{E}[(X-m)]$. By independence the last expectation factors into an inner product of two expectations, each 0; the first two terms are equal (same distribution). Then take the expectation over $Y$.

(c) The optimum of the mean absolute error is the conditional median. For $y = x^2$, $p(x \mid y)$ puts 1/2 on each of $\pm\sqrt{y}$, so any number in $[-\sqrt{y}, +\sqrt{y}]$ is a median — including 0. So switching to the absolute error does not solve the problem: no objective that “outputs one point” can express “both are fine” in the face of two symmetric peaks. The problem is “outputting one point” itself, not the particular error used.

Exercise 2 (the posterior of a generative classifier is a logistic function) A 1-D two-class problem: $p(x \mid y = c) = \mathcal{N}(x;\ \mu_c, \sigma^2)$ (equal variances), prior $p(y=1) = \pi$. Show $p(y = 1 \mid x) = \operatorname{sigmoid}(w x + b)$ and give $w$, $b$. How does the result change if the variances differ? How does this relate to why script 02 gives logistic regression quadratic features?

$\log \frac{p(y=1 \mid x)}{p(y=0 \mid x)} = \log\frac{\pi}{1-\pi} - \frac{(x-\mu_1)^2}{2\sigma^2} + \frac{(x-\mu_0)^2}{2\sigma^2} = \frac{\mu_1 - \mu_0}{\sigma^2}x + \frac{\mu_0^2 - \mu_1^2}{2\sigma^2} + \log\frac{\pi}{1-\pi}$. So $w = (\mu_1 - \mu_0)/\sigma^2$, $b = (\mu_0^2 - \mu_1^2)/(2\sigma^2) + \log\frac{\pi}{1-\pi}$, and $p(y=1\mid x) = \operatorname{sigmoid}(wx+b)$.

With different variances the $x^2$ terms no longer cancel, and the log-odds are a quadratic function of $x$ (a quadratic form in several dimensions). In script 02 the two classes have different covariances, so the generative classifier’s (QDA’s) decision boundary is a quadratic curve. Only with quadratic features does logistic regression’s boundary belong to the same family, which makes the comparison fair.

The exercise also shows that the assumptions of a generative model decide the form of the discriminative model it induces.

Exercise 3 (correcting for a changed prior) During training the class prior is $p(y)$; at test time it becomes $p'(y)$, while the within-class distributions $p(x \mid y)$ do not change. Show that the test-time posterior is $p'(y \mid x) \propto p(y \mid x)\, p'(y)/p(y)$. Why do we say the correction “quietly uses the generative view”?

$p'(y \mid x) = \frac{p(x \mid y)p'(y)}{p'(x)}$ and $p(x \mid y) = \frac{p(y \mid x)p(x)}{p(y)}$. Substituting, $p'(y \mid x) = \frac{p(y\mid x)\,p(x)}{p'(x)} \cdot \frac{p'(y)}{p(y)} \propto p(y \mid x)\frac{p'(y)}{p(y)}$ ($p(x)/p'(x)$ is a constant for a given $x$). The key step of the derivation is ”$p(x \mid y)$ does not change” — an assumption about how data are produced from classes, i.e. a generative assumption. A purely discriminative model does not itself know that “the prior changed but the within-class distributions did not”.

Exercise 4 (δ functions and the training likelihood) For the kernel density estimate $q_h$ of §4.5, show that the mean training log-likelihood $\frac{1}{N}\sum_i \log q_h(x_i) \to +\infty$ as $h \to 0$ (with distinct training points). And for a new point $x$ not in the training set, $\log q_h(x) \to ?$

$q_h(x_i) = \frac{1}{N}\sum_j \mathcal{N}(x_i; x_j, h^2) \ge \frac{1}{N}\mathcal{N}(x_i; x_i, h^2) = \frac{1}{N\sqrt{2\pi}h}$. So $\log q_h(x_i) \ge -\log N - \frac12\log(2\pi) - \log h \to +\infty$. For a new point $x$ at distance $d > 0$ from the nearest training point, $q_h(x) \le \frac{1}{\sqrt{2\pi}h}e^{-d^2/(2h^2)}$, so $\log q_h(x) \le -\frac{d^2}{2h^2} - \log h + \text{const} \to -\infty$ ($1/h^2$ grows much faster than $\log(1/h)$). This is why the held-out likelihood collapses to −99 at $h = 0.005$ in the table.

Exercise 5 (non-negativity of KL) Use Jensen’s inequality ($\log$ is concave, so $\mathbb{E}[\log Z] \le \log \mathbb{E}[Z]$) to prove $D_{\text{KL}}(p \Vert q) \ge 0$. When does equality hold? Deduce that “the floor of the NLL is $H(p_{\text{data}})$”.

$-D_{\text{KL}}(p\Vert q) = \mathbb{E}_p\left[\log\frac{q(x)}{p(x)}\right] \le \log \mathbb{E}_p\left[\frac{q(x)}{p(x)}\right] = \log\int q(x)\,dx = \log 1 = 0$. Equality holds iff $q(x)/p(x)$ is almost everywhere constant on the support of $p$, which together with both integrating to 1 means $p = q$ (almost everywhere). From $\text{NLL} = H(p_{\text{data}}) + D_{\text{KL}} \ge H(p_{\text{data}})$ the floor follows, attained only when $p_\theta = p_{\text{data}}$.

Exercise 6 (the forward-KL-optimal Gaussian is moment matching) Let $q = \mathcal{N}(\mu, \sigma^2)$ and $p$ any 1-D distribution with mean $m$ and variance $v$. Show that $\min_{\mu,\sigma} D_{\text{KL}}(p \Vert q)$ is attained at $\mu = m$, $\sigma^2 = v$. For $p = 0.5\,\mathcal{N}(-a, 1) + 0.5\,\mathcal{N}(a, 1)$, write the optimal $\mu, \sigma^2$, and compare with the result of script 05 (where $p = 0.6\,\mathcal{N}(-2, 0.6^2) + 0.4\,\mathcal{N}(2.5, 0.8^2)$).

$D_{\text{KL}}(p\Vert q) = -H(p) - \mathbb{E}_p[\log q(x)] = -H(p) + \frac12\log(2\pi\sigma^2) + \frac{\mathbb{E}_p[(x-\mu)^2]}{2\sigma^2}$. Since $\mathbb{E}_p[(x - \mu)^2] = v + (m - \mu)^2$, minimizing over $\mu$ gives $\mu = m$. Substituting back and differentiating in $\sigma^2$: $\frac{1}{2\sigma^2} - \frac{v}{2\sigma^4} = 0$, so $\sigma^2 = v$.

For the symmetric two peaks, $m = 0$ and $v = 1 + a^2$, so the optimal $q = \mathcal{N}(0, 1+a^2)$ — spanning both peaks exactly.

Script 05: $m = 0.6\times(-2) + 0.4\times 2.5 = -0.2$; $v = 0.6(0.36 + 4) + 0.4(0.64 + 6.25) - 0.04 = 2.616 + 2.756 - 0.04 = 5.332$, so $\sigma = \sqrt{5.332} \approx 2.309$. This matches the table’s $\mu = -0.200$, $\sigma = 2.309$.

Exercise 7 (the parameter count of the chain rule) How many free parameters does a full probability table for the joint distribution of $d$ binary variables need? What about an autoregressive factorization with every conditional $p(x_i = 1 \mid x_{<i})$ also stored as a full table? And with every conditional a logistic regression $\operatorname{sigmoid}(w_i^\top x_{<i} + b_i)$? (This is the problem the Bengio & Bengio 1999 paper of Lecture 6 sets out to solve.)

• Full joint table: $2^d - 1$ (the probabilities of all combinations, minus the “sums to 1” constraint).
• Autoregressive + full conditional tables: the $i$-th conditional needs one number per value of $x_{<i}$, for a total of $\sum_{i=1}^d 2^{i-1} = 2^d - 1$. The chain rule itself involves no approximation, so the count is the same as the full table.
• Logistic-regression conditionals: the $i$-th has $i-1$ weights plus 1 bias, for a total of $\sum_i i = d(d+1)/2$. That is down from exponential to quadratic, at the price of assuming the form of the conditionals (the “approximation brought by inductive biases” of §6.2).

Neural networks with shared weights can strike a better balance between the two.

Exercise 8 (writing tasks as $p(x \mid y)$) For each task below, give $x$ and $y$, say which carries more information, and whether $H(x \mid y)$ is roughly “large”, “medium” or “close to 0”: (a) machine translation; (b) weather forecasting (given the last hour of radar maps, predict the next hour); (c) speech recognition; (d) text-to-speech; (e) colorizing black-and-white photos.

(a) $y$ = the source sentence, $x$ = the target sentence. Similar amounts of information, but one meaning has several plausible translations: $H(x \mid y)$ medium.

(b) $y$ = past radar maps, $x$ = future radar maps. $x$ carries more information (the details of the future cannot be fully determined by the past), and $H(x\mid y)$ grows with the forecast horizon. This is exactly why Ravuri et al. 2021 used a generative model rather than regression: regression forecasts get blurrier with time.

(c) $y$ = speech, $x$ = text. Speech carries much more information; $H(x\mid y)$ is close to 0 (apart from a little ambiguity such as homophones) — close to a discriminative problem.

(d) $y$ = text, $x$ = speech. The same sentence can be read with countless voices, intonations and rhythms; $H(x \mid y)$ is large — a typical generative problem. Note that (c) and (d) are the two directions of the same pair $(x, y)$.

(e) $y$ = a grayscale image, $x$ = a color image. A piece of clothing can be red or blue: $H(x\mid y)$ medium. Colorizing with L2 regression gives “grayish, brownish” average colors — the color version of the ghosts of §2.

Exercise 9 (hands-on) (a) In 01_one_to_many.py, change the number of neighbours $K$ to 1 and to 50. How do the “crisp” fractions of the nearest-neighbour average and nearest-neighbour sampling change? Explain why. (b) In 06_tiny_deep_generative_model.py, change GaussianConditional to a mixture of three Gaussians (the network outputs 3 sets of $\mu$, $\sigma$ and 3 mixture weights). How low can model D’s bits/dim go? What does this say about the role of the “representation” component?

(a) With $K = 1$, the nearest-neighbour average equals nearest-neighbour sampling (there is only one neighbour); both become “copy the nearest image”, with a crisp fraction close to nearest-neighbour sampling. With $K = 50$, the two digits each make up about half of the 50 neighbours, so the nearest-neighbour average is almost all ghosts (crisp fraction near 0). Nearest-neighbour sampling’s crisp fraction drops slightly, because the neighbours’ styles differ more and the copied bottom halves line up worse. How much you average decides how “blurry” the result is.

(b) Given $x_2$, $x_1$ falls at most at three places on the sine wave, and a mixture of three Gaussians is enough to represent this multimodal conditional. Model D should approach the level of A and B (exact numbers from your own run). This shows that a poorly chosen factorization order can be compensated by a more flexible representation (conditional family); conversely, a good factorization lets a simple representation suffice. “Formulation” and “representation” have to be designed together.

12. Further reading

Works cited on the slides of this lecture

• A. A. Efros & T. K. Leung, Texture Synthesis by Non-parametric Sampling, ICCV 1999. (§1.2; optional reading in Lecture 2)
• C. Barnes, E. Shechtman, A. Finkelstein, D. B. Goldman, PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing, SIGGRAPH 2009. (§1.3; optional reading in Lecture 2)
• W. T. Freeman & J. B. Tenenbaum, Learning Bilinear Models for Two-Factor Problems in Vision. (source of the figure in §4.1)
• J. L. Watson et al., De novo design of protein structure and function with RFdiffusion, Nature 2023. (§1.1, §7.2; required reading in Lecture 23)
• S. Ravuri et al., Skilful precipitation nowcasting using deep generative models of radar, Nature 2021. (§1.1)
• J. Tang et al., LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation, ECCV 2024. (§7.2)
• T. Li et al., Autoregressive Image Generation without Vector Quantization, 2024. (§7.2; required reading in Lecture 8)
• T. Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022. (§7.2; optional reading in Lecture 13)
• C. Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023. (§7.2; required reading in Lecture 21)
• OpenAI, GPT-4 Technical Report, 2023. (§7.2)

Textbooks and surveys

• K. P. Murphy, Probabilistic Machine Learning: Advanced Topics (2023), Part IV “Generation”: a systematic treatment of VAEs, autoregressive models, flows, GANs and diffusion, with unified notation.
• C. M. Bishop & H. Bishop, Deep Learning: Foundations and Concepts (2024): clear chapters on generative models with just the right amount of mathematics.
• I. Goodfellow, Y. Bengio, A. Courville, Deep Learning (2016), Chapter 20: the landscape of deep generative models before the GAN era.
• G. E. P. Box, Science and Statistics, JASA 1976: one source of the famous remark of §5.1.

Blogs (recommended by the original course as examples for final-project blogs)

• Yang Song, Generative Modeling by Estimating Gradients of the Data Distribution (2021).
• Sander Dieleman, Perspectives on diffusion (2023).
• Lilian Weng, Diffusion Models for Video Generation (2024), and her series on VAEs / GANs / flows.

Related notes in this repository (in Chinese)

• MIT6.S184 Lecture 1: building generative models with differential equations, starting from “generation is sampling”.
• CMU10-423: a panorama of generative AI, with introductory notes on VAEs, GANs and diffusion.

13. Next lecture

Lecture 2: Image priors — generative models before deep learning (original reading session · 09/10).

§1 of this lecture showed that the core ideas of generative models appeared as early as 1999. Lecture 2 goes back to the pre-deep-learning era and asks a more specific question: what makes an image “natural”? The three required papers give three answers:

• Zoran & Weiss (2011): a Gaussian mixture prior learned on 8×8 patches (EPLL) gives the best denoising and deblurring of its time — the “prior” is the core of image restoration;
• Zoran & Weiss (2012): why do Gaussian mixtures work so well for natural images? The answer hides in a simple generative process called the “dead leaves model”: occlusion and scale invariance;
• Ulyanov et al. (2018, Deep Image Prior): the architecture of a completely untrained convolutional network is itself a strong image prior.

We will implement EPLL denoising, the dead leaves model, Efros–Leung texture synthesis, PatchMatch, and a Deep Image Prior on a small CNN, using GlyphWorld and synthetic textures.