Lecture 13 · Diffusion in Practice: Guidance, Distillation and High Resolution

Maps to: MIT 6.S978 Deep Generative Models (Fall 2024), Reading · Tue 10/22/2024 · Diffusion Models (student paper session). Required papers:

• J. Ho, T. Salimans, Classifier-Free Diffusion Guidance, NeurIPS 2021 Workshop on Deep Generative Models (arXiv 2207.12598);
• T. Salimans, J. Ho, Progressive Distillation for Fast Sampling of Diffusion Models, ICLR 2022;
• E. Hoogeboom, J. Heek, T. Salimans, simple diffusion: End-to-end diffusion for high resolution images, ICML 2023.

Optional: R. Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (CVPR 2022); T. Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models (EDM, NeurIPS 2022); A. Ramesh et al., Hierarchical Text-Conditional Image Generation with CLIP Latents (unCLIP, 2022); T. Chen, On the Importance of Noise Scheduling for Diffusion Models (2023); S. Dieleman, Diffusion is spectral autoregression (blog, 2024).

Basis: This session consisted of student presentations, with no published slides. The notes are organized around three practical problems of the DDPM built in Lecture 12 — it cannot be steered strongly toward a condition, it needs hundreds of steps, and it does not scale to high-resolution pixels — and the paper that answers each. Background the papers rely on is explained first: continuous-time notation, the ε / x / v parameterizations, DDIM, classifier guidance. Paper claims are labelled “the paper reports / argues”. All experiments were designed for these notes and run on a CPU, with 2-D mixtures (where every score is exact) and the glyph latents of Lecture 12. Several outcomes differed from what I expected and are reported as they came out:

• with clean labels, guidance only made the glyph model worse; it helped once labels were noisy or the model weak;
• guided DDIM overshoots beyond the class rather than sampling the “tilted” distribution;
• x-prediction failed in our MLP for lack of a skip connection;
• one-shot distillation slightly beat progressive distillation on our small problem;
• the second-order Heun sampler lost to Euler below ~17 steps.

Code: code/lecture13/ — 6 scripts + the shared module practical_diffusion.py. They run in 3–110 s. Reading time: 4–5 hours including running the code.

0. How to read this lecture

0.1 Where we are

Lecture 12 built a DDPM from first principles: a forward process that adds Gaussian noise, a network that predicts the noise, and an ancestral sampler that takes T = 1000 small steps back. On glyph latents it worked (valid 0.90). But three things stand between that model and a system like Stable Diffusion:

The optional papers fill in the rest of the modern recipe: latent diffusion (generate in an autoencoder’s latent space), EDM (treat sampling as solving an ODE well, and precondition the network), unCLIP (generate in a CLIP embedding space first), and two views of the noise schedule (Chen; Dieleman).

0.2 Learning goals

1. Use the continuous-time notation $z_t = \alpha_t x + \sigma_t\varepsilon$, log-SNR $\lambda_t$, and convert between ε-, x- and v-prediction; know which loss weighting each implies (§1).
2. Derive classifier guidance and classifier-free guidance from Bayes’ rule; explain label dropout; explain why guided sampling is not sampling $p(x\mid c)\,p(c\mid x)^w$ (§2).
3. Derive progressive distillation’s target from DDIM; explain why ε-prediction breaks it and v-prediction does not (§3).
4. Derive simple diffusion’s schedule shift from average pooling, and list its other ingredients (§4).
5. Explain EDM’s view of samplers as ODE solvers, its preconditioning, and why v-prediction is a special case (§5).
6. Connect the noise schedule to image spectra: why diffusion generates coarse structure first (§4.5).

0.3 Map

1. Background: the language of the three papers

All three papers (and EDM, Chen, simple diffusion’s successors) write diffusion in continuous time with signal and noise scales. This section translates Lecture 12 into that language.

1.1 Continuous time and log-SNR

Time runs from data ($t = 0$) to noise ($t = 1$):

Lecture 12’s discrete $\bar\alpha_t$ is $\alpha_t^2$ here. The one number that describes a noise level is the log signal-to-noise ratio

Two schedules with the same λ at some t corrupt the data identically there; a schedule is just a map $t \mapsto \lambda_t$, and what matters is how much training and sampling time it spends at each λ (Lecture 12 §5.4 already found that the range of λ mattered more than the number of steps).

The papers of this session use the cosine schedule (Nichol & Dhariwal 2021): $\alpha_t = \cos(\pi t/2)$, $\sigma_t = \sin(\pi t/2)$, so $\lambda_t = -2\log\tan(\pi t / 2)$. It runs from $+\infty$ to $-\infty$; implementations clip it. practical_diffusion.logsnr clips to $[-15, 15]$, so at $t = 1$, $\alpha_1 = 5.5\cdot 10^{-4}$ — pure noise to four digits.

1.2 Three things the network can predict

Given $z_t$, any one of three quantities determines the other two:

(SNR $= e^{\lambda} = \alpha^2/\sigma^2$.) The v identities: $\alpha z - \sigma v = \alpha^2 x + \alpha\sigma\varepsilon - \sigma\alpha\varepsilon + \sigma^2 x = x$, and similarly for ε. The last column is Exercise 3.

Two consequences run through this lecture:

• Each parameterization is also a loss weighting. “Predict ε with a plain MSE” (Lecture 12’s $L_{\text{simple}}$) is “predict x with weight SNR”, which ignores the noisiest levels. Choosing what to predict changes which noise levels the training cares about.
• Each divides by something. ε-prediction divides by α — harmless with many small steps, fatal for a model that must jump from pure noise in one step (§3). x-prediction divides by σ (only to get $\hat\varepsilon$) and, more importantly, must output the clean image itself even when the input is already almost clean. v-prediction divides by nothing.

1.3 DDIM: a deterministic sampler

Lecture 12’s ancestral sampler draws $z_s$ from the Gaussian posterior $q(z_s \mid z_t, x = \hat x)$. Song, Meng & Ermon’s DDIM (2021) instead re-uses the predicted noise:

“Keep the same noise, change its amount.” No fresh randomness enters, so the whole sampler is a deterministic map from the starting noise $z_1$ to a sample. It is a discretization of the probability-flow ODE of Lecture 12 §8 — exactly Euler’s method in suitable variables (§5.2, checked in code 05 to 1e-11). A parameter η interpolates between DDIM (η = 0) and the ancestral sampler (η = 1); practical_diffusion.sample implements both.

That determinism is what progressive distillation needs: a deterministic map can be learned by regression.

2. Classifier-free guidance (Ho & Salimans)

2.1 Why a conditional model is not enough

A class-conditional diffusion model trained to maximize likelihood learns $p(x \mid c)$ — all of it. If some training images labelled “dog” are blurry, ambiguous or barely contain a dog, the model reproduces that. For images, people prefer samples that are more typical of the condition than the data is: sharper, more obviously a dog. GANs had truncation for this (Lecture 10): sample the latent from a narrower distribution and trade diversity for fidelity. Diffusion models have no single latent to truncate. Two ideas filled the gap.

2.2 Classifier guidance (Dhariwal & Nichol 2021)

Train, next to the diffusion model, a classifier $p_\phi(c \mid z_t, t)$ on noisy images at every noise level. Bayes’ rule on the noisy marginal gives

So adding the classifier’s gradient to an unconditional score gives a conditional score. Classifier guidance scales it: sample with the score $\nabla\log p_t(z\mid c) + w\,\nabla\log p_\phi(c\mid z)$ (in ε-terms, $\tilde\varepsilon = \varepsilon_\theta(z, c) - w\,\sigma_t\nabla_z\log p_\phi(c\mid z)$, using $\nabla\log p_t = -\varepsilon^*/\sigma_t$ from Lecture 12 §7.2). With w > 0 the samples are pushed toward images the classifier is sure about. Dhariwal & Nichol report that this let diffusion models beat BigGAN’s FID on ImageNet — the paper’s title, “Diffusion Models Beat GANs” — with the same fidelity/diversity dial as truncation.

The drawbacks Ho & Salimans list:

• a second network, trained on noisy inputs, so off-the-shelf classifiers cannot be used;
• the classifier’s input gradient is exactly what an adversarial attack follows, so better Inception scores might partly come from fooling classifier-like metrics rather than from better images.

2.3 The implicit classifier

Bayes’ rule again, solved for the classifier:

The noisy classifier’s gradient is the difference between the conditional and unconditional noise predictions. A conditional and an unconditional diffusion model together are a classifier — the implicit classifier. Substituting it into classifier guidance:

$w = 0$ is the plain conditional model; $w = -1$ the unconditional one; $w > 0$ extrapolates away from the unconditional prediction. (Many implementations write $s = 1 + w$, the “guidance scale”; Stable Diffusion’s default $s = 7.5$ is $w = 6.5$ here.)

Training: label dropout. Both predictions come from one network. During training, replace the label by a special null token ∅ with probability $p_{\text{uncond}}$ (0.1–0.2), so the same network learns $p(x \mid c)$ and $p(x)$. Sampling costs two forward passes per step (batched together in practical_diffusion.predict).

Any parameterization. Because $\hat x$, $\hat\varepsilon$ and $\hat v$ are affine in each other with the same $z$ (§1.2), guiding any one of them with the same $w$ gives the same sampler (Exercise 2).

Code 01, Part 1 checks the Bayes identity with exact scores of a Gaussian mixture (three overlapping 2-D classes): $\nabla\log p_t(c\mid z)$ by finite differences equals $s_c - s_\varnothing$ to within $10^{-9}$ at every noise level. Classifier guidance and CFG are the same vector field; CFG simply gets the classifier for free.

2.4 What guided sampling actually produces

A tempting reading: at $t = 0$ the guided score is $\nabla\log\big[p(x\mid c)\,p(c\mid x)^w\big]$, so guidance samples the class distribution tilted toward confidently classified points. That would be true if the tilt were applied to the clean distribution and then noised. It is not. Guidance tilts every noisy marginal: $p_t(z\mid c)^{1+w}p_t(z)^{-w}$. Raising densities to powers does not commute with adding noise, so this family is not the noising of any one distribution, and following it is not sampling the tilted target.

Code 01, Part 2 measures the difference with exact scores (class 0, 200 steps). Each cell reads tilted target / guided DDIM / guided ancestral:

• Guidance does its job: samples become more class-like and less diverse as w grows — the fidelity/diversity trade-off.
• But it overshoots. The tilted target stays inside the class and narrows only a little. Guided DDIM narrows much more and, at w ≥ 4, pushes samples beyond the class, away from the other two. The guidance term is largest at high noise, where the classes still overlap, and a deterministic sampler never undoes an early push.
• The ancestral sampler overshoots far less: its fresh noise and the score’s pull toward the data act as a corrector at every step. Recent analyses make “CFG is not the tilted distribution” precise (Chidambaram et al. 2024; Bradley & Nakkiran 2024, who describe CFG as a kind of predictor–corrector).

In images, this overshoot is the familiar look of over-guided samples: over-saturated colours, exaggerated features. We will see thicker strokes on glyphs.

Part 3 trains one network with label dropout (0.1) instead of using exact scores. Its guided samples follow the exact ones in confidence and spread (at w = 1: p(c | x) 0.999 vs 0.999, spread 0.373 vs 0.390). The overshoot at w = 4 is smaller (1.6% outside vs 8.1%), because it happens far from the data, where the network is least accurate. The null label alone generates the three classes in proportions 0.326 / 0.334 / 0.340.

2.5 Experiment: guidance on glyphs

Code 02 trains a class-conditional latent diffusion model (v-prediction, cosine schedule, the 64 glyph latents of Lecture 12) with $p_{\text{uncond}} = 0.1$ and samples 200 glyphs per digit with DDIM (64 steps). Besides the Judge’s columns, on-class is the fraction confidently read as the requested digit, and style spread the standard deviation of the ink amount within a digit (real glyphs: 0.0261) — a proxy for the variety of stroke widths and sizes.

Clean labels. I expected the textbook curve — FID first improves with w, then degrades. It did not:

The plain conditional model is already excellent: valid as high as real glyphs, FID 3.7, the requested digit 99.8% of the time. Seven-segment classes never overlap: every real glyph is unambiguously one digit, so $p(c\mid x) = 1$ on the data and the tilted distribution equals $p(x\mid c)$. Guidance has nothing to correct, and only its side effect remains: pushing away from the other digits at high noise thickens the strokes (ink 0.136 → 0.162) and steadily worsens FID. The unconditional branch, trained on 10% of the examples, is much worse than the conditional one.

Noisy labels. Real data are not like that: captions and labels describe images only partly. To mimic this, 30% of the training labels are replaced by random digits (27% end up wrong). Now $p(x\mid c) \approx 0.73\cdot(\text{digit } c) + 0.27\cdot(\text{other digits})$:

At w = 0 a third of the samples are not the requested digit. Guidance re-weights by the implicit classifier $p(c\mid x)^w$, and the wrong digits disappear: on-class reaches 0.997 by w = 2. FID is U-shaped with its best value at w = 0.5, while the style spread falls steadily. This is the trade-off of the CFG paper.

A weak model. An under-fit model (width 128, 1500 steps) shows the same pattern for a different reason: guidance compensates for the model’s own blurriness. With $p_{\text{uncond}} = 0.1$, FID goes 30.8 → 13.5 → 21.9 → 63.3 for w = 0 / 0.5 / 1 / 2, and on-class 0.871 → 0.963 → 0.986 → 0.996. Changing the dropout rate:

0.1 and 0.2 behave alike; 0.5 starves the conditional model (lower validity and on-class at every w).

The lesson. Guidance is a correction for the gap between what the model learned and what we want: noisy conditioning data, or a model too weak to represent $p(x\mid c)$ sharply. Large text-to-image models live in exactly that regime (web captions are noisy, prompts are open-ended), which is why they use large guidance scales. When there is no gap, guidance has only its cost.

2.6 What the paper reports

Ho & Salimans train class-conditional models on ImageNet 64×64 and 128×128 and sweep w and $p_{\text{uncond}}$. They report:

• Inception Score rises steadily with w, while FID is best at a small guidance strength (around w = 0.1–0.3) and worsens beyond — the same U-shape as our noisy-label model;
• $p_{\text{uncond}} \in \{0.1, 0.2\}$ works better than 0.5 — only a small share of the model’s capacity is needed for the unconditional task;
• samples become visibly less diverse and more saturated at large w.

Their conclusion is that a pure generative model, with no classifier anywhere, can produce the quality/diversity trade-off that classifier guidance produced. They also note a limitation: two forward passes per step slow sampling down. Guided distillation (Meng et al. 2023) later folds the guided teacher into a single student.

2.7 Guidance in practice

CFG became the default way to condition diffusion models on text (GLIDE, Imagen, Stable Diffusion, DALL·E 2’s decoder, video models). Common practice adds a few variations:

• negative prompts: replace ∅ by an unwanted condition, so guidance pushes away from it;
• guidance intervals: apply guidance only at intermediate noise levels, because §2.4’s overshoot comes mainly from the noisiest levels;
• rescaling: renormalize the guided prediction to limit saturation.

All are heuristics around the same formula.

3. Progressive distillation (Salimans & Ho)

3.1 The cost problem, and why DDIM helps

Lecture 12’s DDPM needed ~100 steps for good samples (FID 36.7 at 100, 72.7 at 20 on glyphs). Each step is a full network evaluation, so a diffusion model is ~100× more expensive to sample than a GAN of the same size. Better ODE solvers (§5.2) help down to a few tens of steps. Below that, every solver fails for a structural reason: one deterministic step from pure noise can only land on $\hat x(z_1) = \mathbb E[x \mid z_1] \approx \mathbb E[x]$, the mean of the data. The sampler is not wrong; the network is answering a different question (the conditional mean) from the one we need (a map from noise to one particular sample).

DDIM’s determinism suggests the fix. With N steps, DDIM is a fixed function $f_N: z_1 \mapsto x$. If $f_{2N}$ is a good sampler, train a new network whose $N$-step DDIM reproduces $f_{2N}$. That changes what the network predicts at each step: no longer the conditional mean, but whatever $\hat x$ makes the coarse step land where two fine steps would.

3.2 The algorithm

The student starts as a copy of the teacher (sampled with $2N$ steps). For a training example:

1. draw $x$, a student time $t = i/N$ ($i \in \{1, \dots, N\}$), and noise it: $z_t = \alpha_t x + \sigma_t\varepsilon$;
2. run two teacher DDIM steps, $t \to t' = t - \frac{0.5}{N} \to t'' = t - \frac1N$, reaching $z_{t''}$;
3. find the $\tilde x$ for which one DDIM step from $z_t$ lands exactly on $z_{t''}$:

4. take a gradient step on $w(\lambda_t)\,\lVert\tilde x - \hat x_{\text{student}}(z_t)\rVert^2$.

When the student has converged, it becomes the teacher, $N$ is halved, and the process repeats: 1024 → 512 → … → 4 → 2 → 1. (Edge case: when $t'' = 0$ the teacher’s second step already returns its $\hat x$, so $\tilde x$ is that; the formula with $\alpha_{t''} = 1$, $\sigma_{t''} = 0$ agrees.) practical_diffusion.distill_round implements one round.

Why rounds instead of one jump? Each round is a small change to a network that already works. The student starts from the teacher’s weights, and its target differs from what it already does by the error of one halving. The paper argues that this keeps each round cheap, and reports that the whole procedure costs no more than training the original model.

3.3 The parameterization: why ε breaks and v works

At the end of distillation the student’s single step starts from $z_1$ — essentially pure noise, $\alpha_1 \approx 0$ — and must output a good $\hat x$. With ε-prediction, $\hat x = (z - \sigma\hat\varepsilon)/\alpha$: an error δ in $\hat\varepsilon$ becomes an error δ/α in $\hat x$. With our schedule that is a factor 1/α₁ = 1808. As SNR → 0, ε-prediction stops carrying information about x at all: predicting the noise from (almost) pure noise is trivial, and x is the tiny remainder.

The paper considers three alternatives, all well behaved at SNR = 0:

• predict x directly;
• predict x and ε with two output channels, and mix them by the SNR;
• predict v $= \alpha\varepsilon - \sigma x$, which equals $-x$ at $t = 1$ and $\varepsilon$ at $t = 0$, and from which $\hat x = \alpha z - \sigma\hat v$ divides by nothing.

They pair these with a loss weighting expressed in x-space: “truncated SNR”, $\max(\mathrm{SNR}, 1)\,\lVert x - \hat x\rVert^2$, or “SNR + 1”, which is exactly v-prediction’s plain MSE (§1.2). The paper reports that these choices train the original (undistilled) model about as well as ε-prediction, and that only they survive distillation. v-prediction has since become standard (Imagen Video, Stable Diffusion 2’s 768-px model, and others).

3.4 Experiment: distillation in 2-D

Code 03 trains teachers on Lecture 9’s ring of 8 Gaussians (std 0.05, radius 2) with each parameterization. It then distills 128 → 64 → … → 1 step, 500 updates per round. “Quality” is the fraction of samples within 3 std of a mode. Each cell reads DDIM with the teacher / the distilled student:

• DDIM collapses below ~8 steps. At 4 steps its samples fall on the straight lines between modes (the figure’s octagon); at 1 step all of them land at the centre of the ring, $\mathbb E[x]$.
• The v-student works at every step count. Even with 1 step it covers all 8 modes with quality 0.81, and its samples stay within 0.09 of the 128-step teacher’s from the same noise. Total distillation cost: 7 × 500 = 3500 updates, less than the teacher’s own 4000.
• ε breaks, as the paper argues, in two places. The ε teacher’s own DDIM drifts already at 64 steps: its samples land 0.87 away from its 128-step ones, and quality collapses below 32 steps — the first step multiplies the ε error by 1808. Distillation rescues it down to 16 steps (0.864) and fails from 8 steps on.
• x failed for a reason I had not planned. Its teacher was already poor (quality 0.28), with plain MSE and with the paper’s truncated-SNR weighting alike. An x-predicting network must output the clean point even when its input is already almost clean. Near t = 0 it must reproduce its own input to within the modes’ width (0.05), and an MLP without a skip connection does not. v-prediction has that skip built in: $\hat x = \alpha z - \sigma\hat v \to z$ as $t \to 0$. This is the idea EDM formalizes as preconditioning (§5.2). The paper’s U-Nets, whose skip connections make the identity map easy, distill fine with x.

3.5 Experiment: distilling the glyph generator

Code 04 distills script 02’s clean-label glyph model (64 DDIM steps, FID 3.52 from this set of noises) in six rounds of 1000 updates:

One-step DDIM produces the average glyph of each digit. It is recognizable, so validity stays at 0.993, but all variety is gone (style spread 0.0025, FID 226). The one-step student keeps the teacher’s variety at FID 6.3: 64× cheaper sampling for ~3 FID points. Six rounds took 43 s, against 29 s to train the teacher.

One-shot distillation for comparison. Luhman & Luhman (2021) proposed the obvious alternative: run the full teacher on many noises once, then regress a one-step student on the (noise → sample) pairs. With 20,000 pairs (1.3 M teacher evaluations; progressive used 1.5 M) and 6000 updates, the one-shot student reaches FID 4.10, valid 0.978 — slightly better than the progressive one. On this small problem, the teacher’s map from noise to latents is smooth enough to regress in one go, and the single regression avoids six rounds of accumulated error.

The paper’s argument for progressive distillation is about cost at scale. Its teachers use up to 8192 steps, so one training pair costs 8192 network evaluations, and a one-shot student starts far from its target map. Our experiment does not test that regime.

3.6 What the paper reports, and what came next

The paper distills models on CIFAR-10, ImageNet 64×64 and LSUN bedrooms and churches (128×128), starting from samplers with up to 8192 steps. It reports, for example, FID 3.0 on CIFAR-10 with 4 steps, and quality degrading gracefully down to 1–2 steps.

Limitations:

• the rounds are sequential;
• errors accumulate over rounds;
• a guided teacher needs extra care (Meng et al.’s guided distillation distills the guided prediction, with w as an input).

Later few-step methods remove the round structure: consistency models (Lecture 18) learn a map from any point of the ODE trajectory to its end; adversarial and distribution-matching distillation add a GAN-like loss (Lecture 10 §5).

4. simple diffusion (Hoogeboom, Heek & Salimans)

4.1 The problem

In 2022, high-resolution image diffusion took one of two detours:

• cascades — a 64 × 64 model followed by super-resolution diffusion models (Imagen, DALL·E 2);
• latent diffusion — diffuse in a compressed autoencoder space (Stable Diffusion; §5.1).

Both add components, training stages and failure modes. simple diffusion asks: what does it take to train one diffusion model directly on 256–512 px pixels and match them? Its answer is a short list of modifications; the most general one is about the noise schedule.

4.2 The noise schedule must depend on the resolution

Take a noisy image $z_t = \alpha_t x + \sigma_t\varepsilon$ at resolution $d$ and average-pool it by a factor $k$:

Averaging $k^2$ independent noise values divides their variance by $k^2$. A natural image is smooth at the scale of a few pixels, so $\mathrm{pool}_k(x)$ is essentially the same image at resolution $d/k$. The pooled noisy image is therefore a low-resolution image at SNR $k^2$ times higher:

The global structure of an image lives in its low-resolution version. So at the same t, a 512-px image’s global structure is $(512/64)^2 = 64$ times less noisy than a 64-px image’s. With a schedule designed for 64 px, the 512-px model sees its composition destroyed only at the very end of the forward process. In the reverse direction, it must decide the composition in the first few steps and spends nearly all its time on details. The fix shifts the whole schedule toward noise:

For $d = 512$, every SNR is divided by 64. The paper also proposes an interpolated schedule — moving from the shifted schedule at high noise toward a less shifted one at low noise — to keep enough training at the levels that matter for fine detail. Chen (2023, §5.3) reaches the same conclusion from a different direction.

4.3 Experiment: the same glyph at three resolutions

Training pixel-space models at 32–64 px is beyond our CPU budget (Lecture 12 §5.3), so code 06 tests the claim on the forward process directly.

• Data. It renders the same 2000 glyphs at 16, 32 and 64 px: the same shapes, sampled more finely (HiResGlyphWorld).
• Noising. It noises them with the cosine schedule, with and without the shift (shift = d/16; our base resolution is 16, the paper’s 64).
• Measurement. It asks how often the exact Bayes classifier — the posterior over digits under an empirical prior of 5000 glyphs, applied after pooling to 16 px — still recognizes the digit:

At t = 0.8 the digit is recognizable in a third of the 16-px glyphs and in 90% of the 64-px ones. With the unshifted schedule, a 64-px model must settle which digit within the first ~10% of the reverse process (accuracy is already 0.62 at t = 0.9); at 16 px the digit forms gradually between t ≈ 0.9 and 0.6. The shift $-2\log(d/16)$ puts all three resolutions back on the same curve, to within sampling noise.

4.4 The other ingredients

The paper’s remaining modifications are about where to spend compute in a U-Net at high resolution:

• Scale the low-resolution part. Put the extra blocks (and, in their U-ViT variant, transformer layers) at the 16 × 16 level, where global structure is computed and feature maps are cheap. Keep the high-resolution levels thin.
• Downsample early. Use an invertible downsampling in the first layer (a discrete wavelet transform or a patchify), so no full-resolution feature maps are needed.
• Dropout only at low resolution. Regularize where the capacity is.
• Multiscale loss. Add the loss evaluated on downsampled versions of the prediction, to weight the low frequencies — the global structure — more.

The paper reports that a single pixel-space model trained this way reaches FIDs on ImageNet 128–512 competitive with cascaded and latent models. It also reports that the same recipe works for text-to-image generation.

4.5 The spectral view: diffusion generates coarse structure first

The pooling argument has a frequency-domain twin (Dieleman, Diffusion is spectral autoregression). The power spectrum of natural images falls roughly as $1/f^2$; Gaussian noise has a flat spectrum. A noisy image’s spectrum is approximately signal plus a constant noise floor, and the floor rises with t. So the forward process drowns frequencies from the top down: fine detail disappears first, global layout last. The reverse process then restores them from the bottom up — an approximate autoregression over frequencies, coarse to fine.

Code 06, Part 2 computes the highest frequency (cycles per image) at which the mean signal power still exceeds the noise floor:

The dead leaves images of Lecture 2 have a power spectrum ∝ $f^{-2.15}$, close to natural images. Their cutoff descends smoothly from 13 cycles at t = 0.3 to 0 at t = 0.9: a coarse-to-fine schedule over frequencies. The glyphs at 64 px keep more cycles per image than at 16 px at every t, and the shifted schedule brings them back exactly onto the 16-px cutoffs. This is §4.3 in the frequency domain: a larger image has more signal power per frequency band relative to the per-pixel noise.

The view also explains Lecture 12’s “shared intermediate latents” (coarse content decided first) and why the schedule — how long the sampler spends at each cutoff — matters so much.

5. Optional readings

5.1 Latent diffusion (Rombach et al. 2022)

The division of labour. Most bits of an image are imperceptible high-frequency detail. Rombach et al. separate perceptual compression from semantic compression:

• an autoencoder (trained once with a perceptual and a patch-adversarial loss, and a light KL or VQ penalty — the VQ-GAN recipe of Lecture 9) maps images to a latent grid $f$ times smaller per side;
• the diffusion model only has to model the latent grid.

The paper reports that $f = 4$–$8$ gives the best trade-off: smaller factors leave the diffusion model too much work; larger ones lose detail in the autoencoder. Conditioning on text enters through cross-attention. Stable Diffusion is this design.

We have used this idea since Lecture 12 §5.3: a pixel-space MLP DDPM gave valid 0.44 on glyphs, the same algorithm on Lecture 8’s 64 latent numbers 0.90. Every glyph experiment of this lecture runs in that latent space. Latent diffusion and simple diffusion are two answers to the same question. One moves the problem to a space where it is small; the other keeps pixels and fixes the schedule and the architecture so that the model spends its effort where the information is.

5.2 EDM: the design space (Karras et al. 2022)

EDM rewrites every diffusion model in one form: $x = x_0 + \sigma n$ (variance exploding; the VP form is the same after dividing by α, $\sigma = \sigma_{\text{VP}}/\alpha$). The model is a denoiser $D(x; \sigma) \approx \mathbb E[x_0 \mid x]$. Its probability-flow ODE is

EDM then treats the pieces as independent design choices.

Sampling as ODE solving.

• DDIM is Euler’s method on this ODE. Code 05, Part 1 checks it: DDIM in VP variables and Euler on EDM’s ODE agree to $2\cdot 10^{-11}$.
• Heun’s second-order method. Take an Euler step, re-evaluate the slope at the new point, and use the average. It costs 2 evaluations per step instead of 1 (except the last step).
• The time grid. $\sigma_i = \big(\sigma_{\max}^{1/\rho} + \tfrac{i}{N-1}(\sigma_{\min}^{1/\rho} - \sigma_{\max}^{1/\rho})\big)^\rho$ with ρ = 7, which spends most steps at small σ, where trajectories bend.
• Optional stochasticity (“churn”): add a little noise and denoise again, as a corrector.

The paper reports CIFAR-10 FIDs of 1.79 (class-conditional) and 1.97 (unconditional) with 35 network evaluations.

Code 05 isolates the sampler from the network with an exact denoiser (the 8-mode ring is a Gaussian mixture), so every error is discretization error. Mean distance to the exact ODE solution from the same noise:

• At large budgets EDM’s choices win. Euler’s error falls roughly as 1/NFE, Heun’s much faster; at 65 NFE, Heun on the ρ = 7 grid is 3–4× more accurate than any Euler variant.
• At small budgets they lose. At ≤ 17 NFE, Heun is far worse than Euler: its corrector evaluates the slope after a huge step, where it means nothing. And no time grid wins at every budget. A better ODE solver does not give few-step generation; §3 and Lecture 18 do.
• The grid also changes the spread of the samples. The cosine grid ends with one large jump (σ = 0.2 → 0.002 in the 9-level grid). Its last step replaces x by $D(x;\sigma) = \mathbb E[x_0 \mid x]$, which is tighter than the data, so samples are pulled onto the mode centres (spread 0.03–0.04 instead of the data’s 0.05). This happens with the deterministic and the ancestral sampler alike — Lecture 8 §2.4’s tightening, from the sampler’s side.

Preconditioning. EDM writes the network as

with $c_{\text{in}} = 1/\sqrt{\sigma^2 + \sigma_d^2}$ (unit-variance input), $c_{\text{skip}} = \sigma_d^2/(\sigma^2 + \sigma_d^2)$, $c_{\text{out}} = \sigma\sigma_d/\sqrt{\sigma^2 + \sigma_d^2}$ (unit-variance target for $F$), loss weight $1/c_{\text{out}}^2$, and $\sigma_d$ the data’s standard deviation. These are chosen so that the network’s input and target both have unit variance at every σ, and so that $F$‘s errors are amplified as little as possible.

Translate to VP variables with $\sigma_d = 1$ ($x = z/\alpha$, $\sigma \to \sigma/\alpha$; Exercise 7):

• $c_{\text{in}}x = z$;
• $c_{\text{skip}}x = \alpha z$;
• $c_{\text{out}} = \sigma$;
• the loss weight is $1/\sigma^2 = 1 + \mathrm{SNR}$.

So $D = \alpha z + \sigma F$ is v-prediction ($F = -\hat v$) with v’s plain MSE. The parameterization question of §3.3 and code 03 is a preconditioning question:

• ε-prediction has $c_{\text{skip}} = 1/\alpha$, $c_{\text{out}} = -\sigma/\alpha$, which blows up at high noise;
• x-prediction has $c_{\text{skip}} = 0$, so the network must reproduce its input at low noise;
• v (EDM with $\sigma_d = 1$) is bounded everywhere.

EDM also samples training noise levels from a log-normal around the levels that matter most. That is a schedule choice again.

5.3 Noise schedules and input scaling (Chen 2023)

Chen makes the same observation as simple diffusion — the optimal schedule depends on image size — from a different angle. Scaling the input by $b < 1$ ($z_t = \alpha_t b x + \sigma_t\varepsilon$) shifts the log-SNR by $2\log b$. So “use a noisier schedule” and “scale the image down” are the same intervention. Chen proposes simple families of schedules (cosine, sigmoid) plus input scaling, and reports that with them a pixel-space model (the RIN architecture) trains well at high resolution in a single stage. Together with simple diffusion and Lecture 12 §5.4’s T = 500 result, the lesson is consistent: where on the log-SNR axis training happens matters more than how many steps the chain has.

5.4 unCLIP (Ramesh et al. 2022, “DALL·E 2”)

unCLIP generates in two stages through CLIP’s joint image–text embedding space:

• a prior maps a caption to a CLIP image embedding. The paper tries an autoregressive prior and a diffusion prior, and reports that the diffusion prior is comparable or better and cheaper;
• a decoder — a diffusion model, followed by two super-resolution diffusion models — generates the image from that embedding.

Because the decoder is conditioned on an image embedding, an existing image can be encoded and decoded again into variations that keep its semantics, and two images can be interpolated in embedding space. Classifier-free guidance is used in both stages. The paper reports greater diversity than GLIDE at similar photorealism in human evaluations. It is a cascade and a latent model at once: the “latent” is a semantic embedding rather than a compressed image.

5.5 Diffusion is spectral autoregression (Dieleman 2024)

This is §4.5. The blog’s measurement is the radially averaged power spectral density of natural images. On a log–log plot it is close to a straight line (a power law), and noise adds a horizontal floor that rises with the noise level. Its argument is that diffusion is approximately autoregression over frequencies — approximately, because the cutoff is soft and the same network handles all frequencies jointly. That is why the analogy to Lecture 5’s autoregression is a lens, not an identity. It also explains why schedules tuned for one image size (one spectrum) fail at another.

6. The big picture

6.1 Three fixes, one recurring theme

The recurring theme is the noise-level axis:

• guidance acts mostly at high noise;
• distillation fails where α → 0;
• schedules decide how much time each frequency band gets;
• samplers decide where to put their steps.

Choosing what happens at each log-SNR is most of the craft of diffusion models.

6.2 How they combine in practice

A 2023–2024 text-to-image system typically has these parts:

• latent diffusion: an autoencoder with $f = 8$;
• a v- or flow-matching parameterization (Lecture 16);
• a resolution-dependent schedule shift (Stable Diffusion 3 shifts its timesteps by resolution, Lecture 17);
• classifier-free guidance with a scale of 3–8;
• a solver with 20–50 steps, or a distilled few-step student (consistency or adversarial distillation, Lectures 10 and 18) when speed matters.

Each of the papers of this session supplied one of these parts.

7. Summary

1. Notation. $z_t = \alpha_t x + \sigma_t\varepsilon$, log-SNR $\lambda_t$. ε-, x- and v-prediction are interchangeable given $z_t$, but each implies a loss weighting (SNR, 1, SNR + 1) and divides by something different (α, σ, nothing).
2. Classifier-free guidance. Bayes’ rule makes $\varepsilon_c - \varepsilon_\varnothing$ an implicit classifier gradient; one network with label dropout gives both. In 2-D, guidance trades diversity for class-likeness but overshoots beyond the class (57% outside at w = 8 with DDIM). On glyphs it was useless with clean labels (FID 3.7 → 117 by w = 4) and helpful with noisy labels (FID 8.2 → 2.9 at w = 0.5, on-class 0.68 → 0.89).
3. Progressive distillation. Two teacher DDIM steps become one student step, and the rounds repeat. With v-prediction a 1-step student covers all 8 modes of the ring (DDIM: 0) and reaches FID 6.3 on glyphs (DDIM: 226). ε-prediction breaks as α → 0.
4. simple diffusion. Pooling by k raises the effective log-SNR by 2 log k, so high-resolution models need a noisier schedule. At t = 0.8 the digit is recognizable in 34% of 16-px glyphs and in 90% of 64-px ones; the shift brings 64 px to 31%.
5. EDM. DDIM = Euler; Heun with ρ = 7 is best at 33–65 NFE and worst at ≤ 17. Preconditioning with σ_d = 1 is v-prediction.
6. Spectra. Noise drowns frequencies from the top down, so diffusion generates coarse structure first.

8. Common pitfalls

1. “Guidance always improves samples.” It corrects a gap between the model’s conditional distribution and the one we want. With clean, non-overlapping classes and a good model, every w > 0 made FID worse (§2.5).
2. “CFG samples $p(x\mid c)p(c\mid x)^w$.” It tilts each noisy marginal, which is not the same; deterministic guided samplers overshoot beyond the data (§2.4).
3. “w and the guidance scale are the same number.” $s = 1 + w$; papers and libraries differ.
4. “A better ODE solver makes one-step sampling possible.” One step from noise can only return the conditional mean; second-order solvers are worse than Euler at very few steps (§5.2). Few-step sampling needs a network trained for it (§3, Lecture 18).
5. “ε-, x- and v-prediction are equivalent.” Mathematically interchangeable, numerically not: they weight the noise levels differently and amplify errors differently (§1.2, §3.4).
6. “The schedule that works at 64 px works at 512 px.” The same t is far less noisy for a larger image; shift it by $2\log(64/d)$ (§4.2).
7. “Distillation is always cheaper than direct regression.” At our scale one-shot distillation was as good and as cheap; the advantage of progressive rounds is about expensive teachers (§3.5).

9. Exercises

Exercise 1 (the implicit classifier). Show $\nabla_z\log p_t(c\mid z) = -\frac{1}{\sigma_t}(\varepsilon^*(z,c) - \varepsilon^*(z))$, and that the CFG prediction corresponds to the score $s_\varnothing + (1 + w)(s_c - s_\varnothing)$.

By Bayes, $\log p_t(c\mid z) = \log p_t(z\mid c) - \log p_t(z) + \log p(c)$; the last term does not depend on z. By Tweedie (Lecture 12 §7.2), $\nabla\log p_t(z) = -\varepsilon^*(z)/\sigma_t$ and $\nabla\log p_t(z\mid c) = -\varepsilon^*(z,c)/\sigma_t$, where $\varepsilon^*$ is the posterior mean of the noise. Subtract.

For the second part, $\tilde\varepsilon = (1+w)\varepsilon_c - w\varepsilon_\varnothing$ gives the score $-\tilde\varepsilon/\sigma_t = (1+w)s_c - w s_\varnothing = s_\varnothing + (1+w)(s_c - s_\varnothing)$. Equivalently, it is $s_c + w\,\nabla\log p_t(c\mid z)$ — classifier guidance with the implicit classifier.

Exercise 2 (guide in any parameterization). Show that guiding $\hat x$, $\hat\varepsilon$ or $\hat v$ with the same w gives the same DDIM update.

For fixed z and t, each conversion in §1.2 is affine: e.g. $\hat x = (z - \sigma\hat\varepsilon)/\alpha$. An affine map $g(u) = a + Bu$ commutes with combinations whose weights sum to 1: $g((1+w)u_c - wu_\varnothing) = (1+w)g(u_c) - wg(u_\varnothing)$, because the constant term appears with total weight $(1 + w) - w = 1$. So guiding ε and then converting equals converting and then guiding x (or v). DDIM’s update is affine in $(\hat x, \hat\varepsilon)$, so the samplers agree.

Exercise 3 (parameterizations as weightings). Show $\lVert\varepsilon - \hat\varepsilon\rVert^2 = \mathrm{SNR}\,\lVert x - \hat x\rVert^2$ and $\lVert v - \hat v\rVert^2 = (1 + \mathrm{SNR})\lVert x - \hat x\rVert^2$.

ε. $\varepsilon = (z - \alpha x)/\sigma$ and $\hat\varepsilon = (z - \alpha\hat x)/\sigma$, so $\varepsilon - \hat\varepsilon = -\frac{\alpha}{\sigma}(x - \hat x)$. Square it: the factor is $\alpha^2/\sigma^2 = \mathrm{SNR}$.

v. $x = \alpha z - \sigma v$ and $\hat x = \alpha z - \sigma\hat v$, so $x - \hat x = -\sigma(v - \hat v)$, i.e. $\lVert v - \hat v\rVert^2 = \lVert x - \hat x\rVert^2/\sigma^2$. Finally, $1/\sigma^2 = (\alpha^2 + \sigma^2)/\sigma^2 = 1 + \mathrm{SNR}$.

Consequence. ε-prediction gives the noisy levels (SNR → 0) almost no weight in x-space. v keeps weight ≥ 1 everywhere — “SNR + 1” in the paper’s table.

Exercise 4 (the distillation target). Derive $\tilde x$ in §3.2, and check that with $t'' = 0$ ($\alpha_{t''} = 1$, $\sigma_{t''} = 0$) it equals the teacher’s second $\hat x$.

One DDIM step from $z_t$ with prediction $\tilde x$ lands at $\alpha_{t''}\tilde x + \sigma_{t''}\frac{z_t - \alpha_t\tilde x}{\sigma_t}$. Setting this equal to $z_{t''}$ and collecting $\tilde x$ gives $\tilde x\,(\alpha_{t''} - \frac{\sigma_{t''}}{\sigma_t}\alpha_t) = z_{t''} - \frac{\sigma_{t''}}{\sigma_t}z_t$.

At $t'' = 0$: $\tilde x = z_{t''}$, which is the teacher’s second step’s output, $\hat x_{\text{teacher}}(z_{t'})$.

The denominator vanishes only if $\alpha_{t''}/\sigma_{t''} = \alpha_t/\sigma_t$, i.e. $t'' = t$, which never happens.

Exercise 5 (DDIM is Euler). Show that DDIM’s update, written for $x = z/\alpha$ and $\sigma_{\text{VE}} = \sigma/\alpha$, is Euler’s method for $dx/d\sigma = (x - D)/\sigma$.

DDIM. Divide $z_s = \alpha_s\hat x + \sigma_s\frac{z_t - \alpha_t\hat x}{\sigma_t}$ by $\alpha_s$:

$x_s = \hat x + \frac{\sigma_s/\alpha_s}{\sigma_t/\alpha_t}\,(x_t - \hat x) = \hat x + \frac{\sigma^{\text{VE}}_s}{\sigma^{\text{VE}}_t}(x_t - \hat x).$

Euler. $x_s = x_t + (\sigma_s - \sigma_t)\frac{x_t - D}{\sigma_t} = D + \frac{\sigma_s}{\sigma_t}(x_t - D)$.

These are the same, with $D = \hat x$. Code 05 checks it numerically to $2\cdot 10^{-11}$.

Exercise 6 (the shift). Derive the shift from average pooling. How large is it for 256 px relative to 64 px, in log-SNR and in SNR? What does simple diffusion’s rule do at d = 32?

Pooling by k divides the noise variance by $k^2$ and leaves a smooth signal unchanged, so the effective SNR of the low-resolution image is multiplied by $k^2$: $\lambda + 2\log k$. To give a $d$-px model the same coarse-scale corruption as a 64-px model at every t, subtract $2\log(d/64)$.

For d = 256: $2\log 4 \approx 2.77$, i.e. every SNR is divided by 16.

At d = 32 the rule adds $2\log 2$: a smaller image needs a less noisy schedule, by the same argument in reverse.

Exercise 7 (EDM preconditioning is v-prediction). With $\sigma_d = 1$, $x = z/\alpha$ and $\sigma_{\text{VE}} = \sigma/\alpha$, show $c_{\text{in}}x = z$, $c_{\text{skip}}x = \alpha z$, $c_{\text{out}} = \sigma$ and $1/c_{\text{out}}^2 = 1 + \mathrm{SNR}$.

$\sigma_{\text{VE}}^2 + 1 = (\sigma^2 + \alpha^2)/\alpha^2 = 1/\alpha^2$. Hence:

• $c_{\text{in}} = \alpha$, so $c_{\text{in}}x = \alpha\cdot z/\alpha = z$;
• $c_{\text{skip}} = \alpha^2$, so $c_{\text{skip}}x = \alpha z$;
• $c_{\text{out}} = (\sigma/\alpha)\cdot\alpha = \sigma$.

So $D = \alpha z + \sigma F$, which is $\hat x = \alpha z - \sigma\hat v$ with $F = -\hat v$. The weight $1/\sigma^2 = 1 + \mathrm{SNR}$ is v’s plain MSE (Exercise 3).

Exercise 8 (why one deterministic step returns the mean). Show that the minimizer of $\mathbb E\lVert x - \hat x(z_1)\rVert^2$ is $\mathbb E[x \mid z_1]$, and that as $\alpha_1 \to 0$ it tends to $\mathbb E[x]$. Why does a distilled student escape this?

The minimizer. The minimizer of a squared error over functions of $z_1$ is the conditional mean.

The limit. As $\alpha_1 \to 0$, $z_1 = \alpha_1 x + \sigma_1\varepsilon$ carries no information about x, so $\mathbb E[x \mid z_1] \to \mathbb E[x]$. That is code 03’s one-step DDIM landing at the ring’s centre, and code 04’s average glyph.

Why the student escapes. Its training target is not x but $\tilde x$, the end point of the teacher’s deterministic trajectory from $z_1$. That is a function of $z_1$, so there is no averaging over x. The student learns a map from noise to samples, as a GAN generator or a normalizing flow does.

Exercise 9 (hands-on). (a) In 02_cfg_glyphs.py, sample the noisy-label model with the ancestral sampler (eta=1.0). Does the FID-optimal w change? (b) In 03_distill_2d.py, give the x-prediction Denoiser a skip connection: output z + net(...) for small t. How close do you get to v? (c) In 06_resolution_spectra.py, add 128 px (k = 8). Predict the accuracy at t = 0.9 before running.

(a) In 2-D the ancestral sampler overshot much less (§2.4), so expect higher w to remain usable: the FID minimum should move to a larger w or become flatter.

(b) A skip $\hat x = c(t)\,z + (\ldots)$ with $c(t) \to 1$ as $t \to 0$ removes the identity-learning problem. Choosing $c = \alpha$ and output scale σ recovers v exactly (Exercise 7).

(c) The shift is $2\log 8 \approx 4.16$ in log-SNR. At t = 0.9 (λ = −3.68) the effective λ at 16 px is ≈ +0.48, between t = 0.3 and 0.5 on the 16-px curve, so expect an accuracy around 0.85–0.9.

10. Further reading

Required papers

• J. Ho, T. Salimans, Classifier-Free Diffusion Guidance, NeurIPS 2021 Workshop (arXiv 2207.12598).
• T. Salimans, J. Ho, Progressive Distillation for Fast Sampling of Diffusion Models, ICLR 2022.
• E. Hoogeboom, J. Heek, T. Salimans, simple diffusion: End-to-end diffusion for high resolution images, ICML 2023.

Optional papers of the session

• R. Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022.
• T. Karras, M. Aittala, T. Aila, S. Laine, Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022.
• A. Ramesh et al., Hierarchical Text-Conditional Image Generation with CLIP Latents, 2022.
• T. Chen, On the Importance of Noise Scheduling for Diffusion Models, arXiv 2023.
• S. Dieleman, Diffusion is spectral autoregression (blog), 2024.

Background and follow-ups

• P. Dhariwal, A. Nichol, Diffusion Models Beat GANs on Image Synthesis, NeurIPS 2021 (classifier guidance).
• J. Song, C. Meng, S. Ermon, Denoising Diffusion Implicit Models, ICLR 2021 (DDIM).
• E. Luhman, T. Luhman, Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed, 2021 (one-shot distillation).
• C. Meng et al., On Distillation of Guided Diffusion Models, CVPR 2023.
• D. Kingma et al., Variational Diffusion Models, NeurIPS 2021 (log-SNR notation).
• M. Chidambaram et al., What does guidance do? A fine-grained analysis in a simple setting, 2024; A. Bradley, P. Nakkiran, Classifier-Free Guidance is a Predictor-Corrector, 2024.

Related notes in this course

• Lecture 10 (truncation; Diffusion2GAN); Lecture 12 (DDPM, Tweedie, the VP-SDE); Lecture 8 §2.4 (few-step tightening); Lecture 2 (dead leaves and natural-image spectra).

11. Next lecture

Lecture 14: Diffusion without Gaussian denoising (reading session · 10/24).

This lecture treated Gaussian noise as given and tuned everything around it. Lecture 14 asks whether the noise is needed at all. Cold diffusion replaces it with deterministic degradations — blur, masking, pixelation — and still restores images step by step. Inverse heat dissipation uses the heat equation, which erases high frequencies first — the spectral view of §4.5, made literal. InDI turns restoration into small direct iterations, trading distortion for perceptual quality. A recurring question will be diversity: a deterministic degradation has a deterministic inverse, so where does the variety of samples come from?