Lecture 5 · 无监督学习 · Unsupervised Learning: k-means, EM, PCA & ICA

对应 Maps to：Stanford CS229 Autumn 2018（Andrew Ng）Lectures 14–16（Expectation-Maximization Algorithms; EM Algorithm & Factor Analysis; Independent Component Analysis 的前半），以及 CS229 main notes（Andrew Ng & Tengyu Ma）Chapters 10–14（clustering & k-means、EM 与变分推断、PCA、ICA、self-supervised learning & foundation models）。Factor analysis 依据 2018 年 L15 的录像和 CS229 的 factor analysis 讲义。 CS229 当前学期（Summer 2026）的课表需要 Stanford 登录才能看，所以本讲以公开的讲义和 2018 年的录像为准；这部分内容多年来基本没变。 配套代码 Code：code/lecture05/（numpy；5 个脚本，最长的约 10 秒） 预计阅读时间：2.5–3 小时（含运行代码）。这是一份复习讲义：假设你学过这些算法，重点放在”为什么”和”它们之间的联系”上。

0. 本讲导航 · How to read this lecture

0.1 Learning goals

After this lecture you should be able to:

1. 说出 k-means、GMM、factor analysis、PCA、ICA 各自的**隐变量（latent variable）**是什么、假设了什么，以及它们之间的”极限关系”：k-means 是 $\sigma \to 0$ 的 GMM，PCA 是 $\sigma \to 0$ 的 probabilistic PCA。
2. Derive EM from Jensen’s inequality in three lines, explain why the log-likelihood never decreases, and recognize EM as coordinate ascent on the ELBO — the same structure as k-means’ coordinate descent on the distortion.
3. 解释几种”MLE 本身是病态的”情形：GMM 的协方差塌缩、$d > n$ 时奇异的样本协方差、factor analysis 的 Heywood case —— 以及为什么修法都是”加先验 / 加结构”。
4. Explain why max-variance and min-reconstruction-error give the same PCA subspace, how (and how not) to choose $k$, and when standardizing features helps or hurts.
5. 说清楚 ICA 为什么需要非高斯的源、为什么白化只完成了一半的工作，以及 CS229 讲义里的 sigmoid 先验什么时候会失败。
6. Place self-supervised learning and VAEs in this picture: a VAE is EM with a learned, approximate E-step.

0.2 默认你已经会的 / 本讲会省略的

0.3 本讲地图 Map

§1  The big picture：没有标签能学什么；隐变量模型；主线继续（EM 最大化似然的下界）
§2  k-means：坐标下降 / 局部最优与 k-means++ / 选 k / 隐含的假设              ← 代码 01
§3  GMM 与 EM：E 步与 M 步 / Jensen 与 ELBO / 一般的 EM / hard EM / 似然塌缩    ← 代码 02
§4  Factor analysis：d > n 时协方差奇异；低秩 + 对角                          ← 代码 03
§5  PCA：最大方差 = 最小重构误差 / SVD / 选 k / 标准化                        ← 代码 04
§6  ICA：鸡尾酒会 / 排列与缩放 / 高斯源不可识别 / 基于似然的 ICA / 源分布的假设  ← 代码 05
§7  Self-supervised learning、foundation models、VAE = 摊销的变分 EM（只有文字）
§8–12 小结、常见误区、练习、延伸阅读、下一讲预告

0.4 Notation（新出现的符号）

0.5 关于数字

讲义里所有实验数字都来自配套代码的实际输出（固定随机种子，同一台机器上重复运行逐字相同；用 python3 -W error 运行，没有数值警告）。数据是合成的：我们知道真实的簇、因子和源，所以能检查算法有没有学对；数字本身不代表任何真实系统。几处结果和编写前预想的不一样（最明显的是 §6.5），讲义按实际结果写，代码 README 的”关于实验结果”逐条记录。

1. The big picture · 大图景

1.1 What can you learn without labels?

Supervised learning has a target $y$ that defines the task. Unsupervised learning has only $\{x^{(i)}\}_{i=1}^n$, so the question becomes: what structure in $x$ is worth modeling? CS229 gives four classical answers:

A fifth answer matters most today: representation learning — instead of hand-specifying the structure, let a model learn useful representations from huge amounts of unlabeled data (§7).

1.2 The thread continues: latent variables and likelihood

Lecture 1’s thread was loss = negative log-likelihood. Most models in this lecture are still maximum likelihood, with one twist — an unobserved variable $z$:

The trouble is the sum inside the log. If $z$ were observed, $\log p(x, z)$ would usually have a closed-form MLE (the GMM reduces to GDA); with $z$ hidden, the log-sum couples all the parameters together. EM’s answer is to maximize a lower bound instead (§3.3). This table is the skeleton of the lecture:

Note the two "$\sigma \to 0$" entries: k-means and PCA are both zero-noise limits of probabilistic models. That explains why they are fast (there are no probabilities to compute) and also what they lose: without a likelihood, you cannot choose $k$ by held-out likelihood (§2.3, §5.1).

1.3 Why does this still matter?

• Every foundation model is pretrained without labels. Next-token prediction, masked modeling and contrastive learning turn unlabeled data into a training signal (§7). “Learn $p(x)$” and “learn a representation $z$ of $x$” are exactly the two goals of this lecture, scaled up.
• k-means and PCA are everywhere in modern pipelines: vector-quantized codebooks (VQ-VAE, audio tokenizers), product quantization in vector databases, whitening and PCA of embeddings, the low-rank assumption behind LoRA, probing which directions a network’s activations actually use.
• The ELBO is the training objective of VAEs and diffusion models. Both maximize a variational lower bound on $\log p(x)$ — the same inequality as §3.3.

2. k-means · k 均值聚类

2.1 The objective and coordinate descent

The CS229 notes present the algorithm first and the objective second; for a review, the reverse order is clearer. k-means minimizes the distortion function

There are two blocks of variables: the assignments $c^{(i)} \in \{1, \dots, K\}$ and the centers $\mu_1, \dots, \mu_K$. Optimizing both jointly is hard (discrete + continuous), but each block alone has an exact solution:

• assignment step: fix $\mu$ and set $c^{(i)} := \arg\min_k \lVert x^{(i)} - \mu_k\rVert^2$;
• update step: fix $c$ and set $\mu_k :=$ the mean of cluster $k$ (the sum of squared distances from a set of points to a single point is minimized at their mean).

This is coordinate descent: no half-step can increase $J$, and there are only finitely many assignments, so the algorithm stops after finitely many steps. 代码 01_kmeans.py 演示 1 用 3×3 网格上的 9 个高斯团（540 个点），从一个随机初始化出发：第 1 轮 $J$ 从 5168 降到 2228，12 轮后停在 986.42（每个半步都用 assert 检查了单调性）。可是按真实的团划分，$J = 533.17$ —— 算法停在了局部最优：一个团里挤了 2 个中心，另一个团没有自己的中心，只好由邻居的中心兼管。

2.2 Local optima and k-means++

$J$ 非凸，求全局最优是 NP-hard。局部最优有多常见？演示 2 在同一份数据上各跑 200 次（$J^\star = 531.31$ 是 400 次里最好的值）：

k-means++ (Arthur & Vassilvitskii, 2007): pick the first center uniformly at random, then sample each new center with probability proportional to $D(x)^2$, where $D(x)$ is the distance from $x$ to the nearest center chosen so far. Points far from the existing centers are likely picks, so the centers spread out. It comes with a guarantee, $\mathbb{E}[J] \le 8(\ln K + 2)\,J_{\text{opt}}$; in the experiment it lowers the initial $J$ by about 40%, needs fewer iterations, and its median run is the best solution.

But it is not a cure: 44% of the runs still end in a local optimum. In practice the recipe is always “k-means++ plus several restarts, keep the lowest $J$” (sklearn’s n_init). With 10 restarts, the success rates of the two initializations are 95% and 100%.

2.3 Choosing k

$J$ 随 $k$ 单调下降（$k = n$ 时 $J = 0$），所以不能”选 $J$ 最小的 $k$“——和第 1 讲”多项式次数越高、训练误差越低”是同一件事。演示 3（每个 $k$ 用 k-means++ 重启 10 次）：

团分得开时有清楚的肘部（elbow）：$k = 9$ 之后每多一个中心几乎没有收益。团重叠时曲线平滑，看不出肘部 —— 真实数据通常更像第二种。Other ways to choose $k$:

• the downstream task (clustering is usually one step in a larger pipeline);
• switch to a GMM, which has a likelihood, and choose $K$ by held-out likelihood or BIC — exactly what §1.2 says the $\sigma \to 0$ limit throws away;
• heuristics such as the silhouette score or the gap statistic — or simply let the application decide (“split users into 5 tiers”).

2.4 What k-means assumes

欧氏距离 + 每簇一个中心，隐含的假设是：每个簇都是半径差不多的球。演示 4 用两份违反假设的数据（重启 10 次取 $J$ 最小的解）：

The key is columns 2 and 3: the wrong answer has the lower $J$. This is not an optimization failure — more restarts or k-means++ cannot help — the objective itself prefers the wrong answer. Each strip has variance 16 along x, far more than the spread created by the gap between the strips, so cutting along x removes much more squared distance. The fix is to relax the assumptions: give every cluster its own covariance (to express “elongated”) and its own weight (to express “unequal sizes”). That is the GMM; §3.5 shows its accuracy on these same two datasets.

3. Gaussian mixtures and EM · 高斯混合模型与 EM

3.1 Latent variables: why the MLE is hard

GMM 的生成过程：先抽成分 $z^{(i)} \sim \text{Categorical}(\phi)$，再抽 $x^{(i)} \mid z^{(i)} = j \sim \mathcal{N}(\mu_j, \Sigma_j)$。对数似然

If we knew the $z^{(i)}$, this would be exactly GDA: $\phi_j$ = fraction of points with label $j$; $\mu_j, \Sigma_j$ = sample mean and covariance of those points — all in closed form. The log-of-a-sum couples the parameters and destroys the closed form; setting the gradient to zero gives equations in which every parameter depends on all the others through the posterior. EM’s idea is the natural one: guess the labels softly, solve the “labeled” problem, repeat.

3.2 The EM algorithm for GMMs

• E-step：对每个 $i, j$ 算后验 $w_j^{(i)} = p(z^{(i)} = j \mid x^{(i)}; \phi, \mu, \Sigma)$ —— Bayes 公式，和 GDA 的预测一模一样。
• M-step：把 $w_j^{(i)}$ 当作”软标签”做 GDA：

代码 02_gmm_em.py 演示 1：3 个成分（权重 0.5 / 0.3 / 0.2，一个细长倾斜、一个小而圆、一个反向倾斜），600 个点，随机初始化：

• The log-likelihood never decreases (all 37 steps are checked with an assert). There is a long “climb” in the middle (iterations 10 and 15 still gain +14 and +11 each), which is typical of EM: convergence is linear, and the more the components overlap, the slower it gets.
• The learned parameters are close to the truth (weights 0.491 / 0.308 / 0.200). Clustering by the posterior argmax gives accuracy 0.982; the Bayes-optimal rule with the true parameters gives 0.983.
• $\ell(\text{EM}) = -2184.7 > \ell(\text{true params}) = -2192.5$: the MLE fits this particular sample, so naturally it beats the parameters that generated it.
• 5 more random restarts all reach the same $\ell$. EM only guarantees a local optimum; these components are fairly well separated, so nothing went wrong here. With heavy overlap or a large $K$, multiple restarts are essential.

3.3 Why EM works: Jensen’s inequality and the ELBO

This is the heart of the lecture, and the derivation is three lines. For any distribution $Q_i$ over $z$,

by Jensen’s inequality ($\log$ is concave, so $\log \mathbb{E}[\cdot] \ge \mathbb{E}[\log \cdot]$). The gap is exactly a KL divergence:

So:

• E-step = set $Q_i = p(z \mid x^{(i)};\theta_t)$. The KL becomes zero: the bound touches $\ell$ at $\theta_t$.
• M-step = maximize $\sum_i \text{ELBO}(x^{(i)}; Q_i, \theta)$ over $\theta$ with $Q$ fixed. Because $Q$ is fixed, the $-\log Q$ term is a constant and we maximize $\sum_i \sum_z Q_i(z)\log p(x^{(i)}, z;\theta)$ — a weighted complete-data MLE. For the GMM this is the weighted GDA above.
• Monotonicity is one line: $\ell(\theta_{t+1}) \ge \text{ELBO}(Q_t, \theta_{t+1}) \ge \text{ELBO}(Q_t, \theta_t) = \ell(\theta_t)$.

演示 2 逐轮验证这个链条（列出 6 轮中的 3 轮）：

前 200 轮里，E 步之后 $|\text{ELBO} - \ell|$ 最大 $9\times10^{-13}$（舍入误差）。在收敛的 $\theta$ 处换别的 $Q$，$\ell - \text{ELBO}$ 和直接算的 $\sum_i \text{KL}(Q_i \Vert \text{posterior}_i)$ 逐位相同：均匀的 $Q$ 差 7581.6，k-means 的硬分配只差 85.0（成分分得开时后验本来就接近 one-hot）。

Picture to keep in mind: EM is coordinate ascent on the two-block function $\text{ELBO}(Q, \theta)$ — exactly isomorphic to k-means’ coordinate descent on $J(c, \mu)$, with $Q$ playing the role of the assignments $c$ and $\theta$ the role of the centers $\mu$.

3.4 The general EM algorithm

Nothing above used the Gaussian form. For any latent-variable model:

repeat until convergence:
    E-step: for each i,  Q_i(z) := p(z | x⁽ⁱ⁾; θ)
    M-step: θ := argmax_θ  Σ_i Σ_z Q_i(z) log p(x⁽ⁱ⁾, z; θ)

A few things worth getting straight:

• When is EM a good fit? When the complete-data log-likelihood $\log p(x, z;\theta)$ has a closed-form MLE (exponential families!) and the posterior $p(z \mid x)$ is computable. GMMs, FA, HMMs (Baum–Welch), missing data and the Dawid–Skene model of multiple annotators all have this structure.
• No step size, constraints for free: the M-step automatically satisfies $\sum_j \phi_j = 1$ and $\Sigma_j \succeq 0$, and there is no learning rate to tune. The price is linear convergence (the more information is missing, the slower), so sometimes direct gradient or quasi-Newton methods are used instead.
• Variants: the M-step only needs to increase the ELBO (generalized EM); when the exact posterior is intractable, pick the closest $Q$ within a tractable family (variational EM) — the road that leads to the VAE (§7.3).
• MAP-EM: put a prior on $\theta$ and maximize $\text{ELBO} + \log p(\theta)$ in the M-step; monotonicity still holds. §3.6 uses this.

3.5 k-means is “hard” EM

把 GMM 限制成：权重相等、所有协方差固定为 $\sigma^2 I$、只学均值。E 步变成 $w_j^{(i)} \propto \exp\big(-\lVert x^{(i)} - \mu_j\rVert^2 / 2\sigma^2\big)$。当 $\sigma \to 0$，最近的中心压倒一切，$w^{(i)}$ 变成 one-hot（“分给最近的中心”），M 步的加权平均变成”簇内平均”——这就是 k-means。演示 3(a) 让这个”软 k-means”和 k-means 从同一组初始中心出发：

• $\sigma \le 0.02$ 时软 k-means 和 k-means 是同一个解。
• 中间的 $\sigma$ 并不单调地逼近：迭代早期中心还没到位时，边界附近的点被”软”地分配，路径不同，最后停在了 k-means 的另一个相邻的不动点上（$\sigma = 0.05$ 时最终的责任已经全是 one-hot，差别仍然存在）。
• $\sigma = 3$ 时三个中心合并成了一个（平均最大责任 = 1/3）。$\sigma$ 像一个”分辨率”：比簇间距还大，就看不出有几个簇。从大 $\sigma$ 慢慢降下来就是 deterministic annealing。

反过来，放开这些限制就修好了 §2.4 的两个失败。演示 3(b) 用完全相同的两份数据（GMM 各重启 10 次取似然最大的）：

3.6 When the likelihood is unbounded: component collapse

GMM 的似然有一个”作弊”的解：把一个成分的均值放在某个数据点上，让它的协方差 → 0。这个点的密度 $\propto 1/\sigma^d \to \infty$，其他点由别的成分照顾，于是 $\ell \to +\infty$。演示 4(a) 在演示 1 学到的 GMM 上再加第 4 个成分（权重 0.01，均值 = 第 0 个数据点，协方差 $\sigma^2 I$）：

$\sigma$ 每缩小 10 倍，$\ell$ 增加 $d\ln 10 = 4.61$，没有上界（$\sigma$ 从 1 到 0.1 时 $\ell$ 先略降：多出来的成分拿走了 1% 的权重，却只解释一小块区域）。所以”GMM 的 MLE”在字面意义上不存在；我们真正想要的是一个”好的”局部最大值。

这在实际运行中会发生吗？演示 4(b)：一团没有任何簇结构的二维高斯噪声，成分数偏多，100 个随机初始化：

第一个塌缩的例子里，那个成分只”拥有” $N_k = 2.000$ 个点，协方差的特征值是 $[3\times10^{-19},\ 0.35]$ —— 它塌到了两个点连成的直线上。一般地，在 $d$ 维空间里，一个成分只要抓住不超过 $d$ 个点就能塌缩，不一定非得是 1 个点。

The fix is MAP: add the penalty $-\frac{\lambda}{2}\sum_k \text{tr}(\Sigma_k^{-1})$ (an inverse-Wishart-type prior). The M-step becomes $\Sigma_k = S_k + (\lambda/N_k)\,I$ — still an exact EM step that monotonically increases the penalized objective — and as $\sigma \to 0$ the penalty goes to $-\infty$ like $1/\sigma^2$, much faster than $\log(1/\sigma)$ grows. sklearn’s reg_covar is a simpler constant floor $\epsilon I$. Other remedies: re-initialize a component when it collapses, restart several times, or use BIC to choose fewer components.

The broader lesson: the MLE itself can be ill-posed. Logistic regression on separable data (Lecture 1, $\lVert\theta\rVert \to \infty$), the sample covariance when $d > n$ (§4) and component collapse here are the same kind of problem — the likelihood blows up at the boundary of the parameter space — and the fix is the same: add a prior, or add structure.

4. Factor analysis (briefly) · 因子分析

4.1 Motivation: when d > n, even “fit a Gaussian” fails

CS229 讲 FA 的动机很朴素：特征比样本多时（$d > n$），连拟合一个高斯分布都做不到。$n$ 个点中心化后只张成 $n - 1$ 维，样本协方差 $S$ 奇异：MLE 想让其余方向的方差 → 0，$\log\det \to -\infty$，训练似然 → $+\infty$，而新数据只要不落在那个子空间里，密度就是 0。

代码 03_factor_analysis.py 演示 1：$d = 30$ 个特征由 3 个因子驱动，只有 $n = 20$ 个训练样本（$S$ 的秩 = 19，Cholesky 失败）。平均每个样本的对数似然：

这是第 1 讲”$X^\top X$ 奇异、$d > n$“在密度估计里的翻版。加 $\epsilon I$（shrinkage）能修好，但对 $\epsilon$ 极其敏感，表里最好的 $\epsilon = 1$ 还是看着测试集挑的（乐观）；对角高斯永远可逆，但丢掉了所有相关性。满协方差有 465 个参数，而训练集一共只有 600 个数。需要一个中间的结构。

4.2 The model: low rank + diagonal

• $\Lambda\Lambda^\top$ (rank $k$) captures the correlations caused by $k$ shared factors; $\Psi$ is each feature’s own noise. That is about $dk + d$ parameters instead of $d(d+1)/2$.
• As long as $\Psi \succ 0$, $\Lambda\Lambda^\top + \Psi$ is positive definite — even when $d > n$.
• EM: in the E-step $z \mid x$ is Gaussian (conditional-Gaussian formulas), with $\mathbb{E}[z \mid x] = \Lambda^\top(\Lambda\Lambda^\top + \Psi)^{-1}(x - \mu)$. In the M-step the $\Lambda$ update is a linear regression (normal equation) of $x$ on $\mathbb{E}[z]$, except that $\mathbb{E}[zz^\top]$ carries an extra $\text{Cov}[z \mid x]$ beyond $\mathbb{E}[z]\mathbb{E}[z]^\top$ — an admission that $z$ was guessed and is uncertain.

4.3 What happened in the experiment

演示 2，每个 $k$ 跑 EM（最多 5000 轮）：

• EM 对每个 $k$ 都单调（同一个 ELBO 论证）。训练似然随 $k$ 单调上升，测试似然在真实的 $k = 3$ 最好。
• 诚实地说：只有 20 个样本时，$k = 3$ 的 FA（−51.17）和看着测试集挑了 $\epsilon$ 的 $S + I$（−50.43）差不多，并没有明显更好。FA 的优势是结构从数据里学出来（哪些特征共享哪些因子、每个特征的噪声多大），不需要事后挑 $\epsilon$；两者都远好于对角高斯（−59.70）。
• $k \ne 3$ 时都出现了 Heywood case：某个 $\Psi_j$ 被推到下限（这个特征被因子”完全”解释），EM 在边界附近收敛极慢（都用满了 5000 轮）。这是”MLE 跑向参数空间边界”的又一个例子，通常提示模型设定或样本量有问题。

演示 3 还验证了两件事。(a) $\Lambda$ 只能确定到一个旋转：$\Lambda R$（$R$ 正交）给出完全相同的似然，因为 $\Lambda RR^\top\Lambda^\top = \Lambda\Lambda^\top$；心理测量里常用 varimax 等方法事后挑一个”好解释”的旋转，ICA（§6）则要从数据里确定它。(b) 后验均值 $\mathbb{E}[z \mid x]$ 是一个好的 3 维表示：在测试集上线性回归真实因子的 $R^2 \approx 0.96$，用真参数也只有 $\approx 0.97$（噪声让 $z$ 不能从 $x$ 完全确定）。

Restricting $\Psi$ to $\sigma^2 I$ gives probabilistic PCA (Tipping & Bishop, 1999); letting $\sigma \to 0$ then turns $\mathbb{E}[z \mid x]$ into “project onto the principal directions” — which is PCA.

5. PCA · 主成分分析

5.1 Two definitions, one answer

PCA has two definitions that look different (data centered, $\Sigma = \frac1n\sum_i x^{(i)}x^{(i)\top}$):

1. Maximum variance: find the unit vector $u$ that maximizes the projected variance $u^\top \Sigma u$. The answer is the top eigenvector of $\Sigma$; the span of the top $k$ eigenvectors maximizes the total projected variance.
2. Minimum reconstruction error: find the $k$-dimensional subspace that minimizes the sum of squared distances from each point to its projection.

They are the same problem because of Pythagoras: $\lVert x\rVert^2 = \lVert \text{projection}\rVert^2 + \lVert\text{residual}\rVert^2$. The left side is fixed, so maximizing the first term minimizes the second. Precisely: reconstructing from the top $k$ principal directions gives a mean squared error equal to the sum of the discarded eigenvalues $\sum_{j > k}\lambda_j$.

代码 04_pca.py 演示 2 逐个 $k$ 验证了这个等式（$d = 10$，真实秩 3，各向同性噪声 $\sigma = 0.5$，500 个训练样本，误差都是每个样本的平均）：

前两行的差 $\le 10^{-14}$。两个推论：

• 测试集上的重构误差也随 $k$ 单调下降 —— 更大的子空间总能把任何点投影得更近（练习 4）。所以”在验证集上算重构误差”选不出 $k$，需要一个有似然的模型（PPCA / FA 的 held-out 似然，§4.3）或别的准则。
• PCA 是去噪器：和干净信号的误差在 $k = 3$ 最小（0.77，带噪声的原始数据是 2.55）。丢掉的方向主要是噪声；$k$ 再大，就把噪声也”重构”回来了。

5.2 Computing it: SVD, not the covariance matrix

对中心化后的数据做瘦 SVD：$X = U\,\text{diag}(s)\,V^\top$，则 $\Sigma = \frac1n X^\top X = V\,\text{diag}(s^2/n)\,V^\top$ —— 主方向是 $V$ 的列，$\lambda_j = s_j^2/n$。演示 1 里两种算法的特征值差 $4\times10^{-15}$。实践中用 SVD：不必构造 $X^\top X$（条件数是 $X$ 的平方，第 1 讲 §2.2），$d \gg n$ 时还可以只算前几个奇异向量（randomized SVD）。

Another view: the PCA subspace is the optimum of a linear autoencoder (encoder $V_k^\top$, decoder $V_k$, squared reconstruction error; Baldi & Hornik, 1989 — another way of stating the Eckart–Young theorem). Replace the encoder and decoder by neural networks and you get a nonlinear autoencoder; add a probabilistic interpretation and you are on the way to the VAE of §7.3.

5.3 Choosing k

演示 1 的特征值谱：

• 前 3 个特征值 ≈ 信号方差 + $\sigma^2$，之后是一个 ≈ $\sigma^2$ 的噪声地板。地板不完全平（0.195–0.295）：有限样本让本来相等的特征值散开（Marchenko–Pastur），$d/n$ 越大散得越开。前 3 个主方向和真实的信号子空间几乎重合（最大主角的余弦 0.9983）。
• “解释 X% 的方差”是最常见的规则，但阈值是任意的：80% → $k = 2$（丢了一个真实的因子），95% → $k = 7$，99% → $k = 10$（把噪声方向也收进来了 —— 7 个噪声方向合起来占 10% 的方差）。
• 看谱的”断崖”（scree plot）更可靠：$\lambda_3/\lambda_4 = 4.58$ 是最大的相邻比值（$\lambda_1/\lambda_2 = 2.05$、$\lambda_2/\lambda_3 = 3.31$，之后都 ≤ 1.12）。但最弱的信号只比噪声稍强时，断崖会消失。有原则的做法是概率模型的 held-out 似然，或者看下游任务。

5.4 Standardize or not?

PCA 找”方差最大”的方向，而方差取决于单位 —— PCA 不是尺度不变的。演示 3：5 个有信息的特征由 2 个隐因子驱动；用前 2 个主成分线性回归两个真实因子，$R^2$ 衡量 PCA 子空间保留了多少因子信息（这个度量和坐标、单位无关）：

• (A) 单位不同时要标准化：一个和因子无关、但数值很大的特征霸占了 PC1。换单位的检验更直接：把 x1 从米换成毫米（×1000），不标准化的 PCA 完全变了（PC1 解释的比例 0.740 → 0.99999），标准化的 PCA 一点不变（特征值差 $2\times10^{-15}$）。
• (B) 单位相同、方差本身有意义时，标准化有害：几个几乎为常数的噪声特征本来对 PCA 毫无影响，标准化把它们的方差从 $10^{-4}$ 放大到 1。甚至在只有有信息特征的数据上，标准化也让 $R^2$ 从 0.94 / 0.92 降到 0.92 / 0.86：这些特征噪声相同、信号方差不同（其中 x4 几乎没有信号），方差大就意味着信噪比高，不标准化时给它们更大的权重恰好是对的。
• 规则：单位不同（米、元、人数）→ 标准化（等价于对相关矩阵做 PCA）；单位相同且方差有意义（像素、同一种传感器的读数、词频）→ 只中心化。拿不准时两种都试，看下游任务。

6. ICA · 独立成分分析

6.1 The cocktail party problem

$d$ people talk at once, and $d$ microphones each record a mixture: $x = As$, where $s \in \mathbb{R}^d$ holds the speakers’ signals (independent components) and $A$ is unknown. The goal is the unmixing matrix $W = A^{-1}$, which recovers $s = Wx$. 05_ica.py uses 3 synthetic sources (sine, sawtooth, Laplace noise) and a random $3\times3$ mixing matrix (condition number 4.0); every microphone hears all 3 sources.

6.2 What can never be recovered: permutation and scale

$x = As = (AP^{-1}D^{-1})(DPs)$ holds for every permutation matrix $P$ and every invertible diagonal matrix $D$, so:

• Permutation: there is no way to know which recovered signal is “speaker 1” — the same issue as label switching in clustering;
• Scale and sign: multiply a source by $c$ and divide the corresponding column of $A$ by $c$, and the observations do not change at all. For audio that is just volume and polarity, which we don’t care about.

No method can remove these two ambiguities, so ICA is evaluated with the $|\text{corr}|$ matrix: perfect separation makes it a permutation matrix.

6.3 Why Gaussian sources are hopeless, and why whitening is only half the job

Start with PCA whitening: $\tilde x = \text{diag}(\lambda)^{-1/2}V^\top(x - \bar x)$, whose covariance is $I$. Demo 1: after whitening the components are uncorrelated (the covariance differs from $I$ by $1.6\times10^{-15}$), yet each component’s largest $|\text{corr}|$ with any source is only 0.71, 0.81, 0.90 — still mixed.

The reason: after whitening, multiplying by any orthogonal $R$ keeps the covariance at $RIR^\top = I$. Second-order statistics are blind to rotations, so the remaining rotation must be found from higher-order statistics. And if the sources are Gaussian, $s \sim \mathcal{N}(0, I)$ is itself rotation-invariant: $Rs$ and $s$ have the same distribution, and no amount of data can tell them apart. This is the identifiability condition of ICA (Comon, 1994): at most one source may be Gaussian. It is also why FA’s $\Lambda$ is only determined up to rotation — FA’s factors are Gaussian.

Whitening still helps a lot: it reduces the unknown from a general $d \times d$ matrix to a rotation ($d(d-1)/2$ degrees of freedom), so practical ICA almost always whitens first.

演示 2 把这件事直接画出来：两个单位方差的独立源，$W = c\,R(\theta)$（$c = \pi/\sqrt3$，使 logistic 先验的方差匹配），计算 CS229 ICA 模型（§6.4）下的平均对数似然，$n = 20000$：

（0° 和 90° 都是正确的分离。）高斯源的似然对旋转是平的（只有抽样波动）；super-Gaussian 的 Laplace 源在正确的角度最大；而 sub-Gaussian 的正弦和锯齿波在 45°——最混合的角度——最大。第三行是本讲最出乎意料的结果，§6.5 解释。

6.4 Likelihood-based ICA (the CS229 version)

ICA is maximum likelihood too. The change-of-variables formula gives $p_x(x) = p_s(Wx)\,|\det W|$, and independence of the sources gives $p_s(s) = \prod_j p_s(s_j)$:

We have to pick a density for $p_s$, and it cannot be Gaussian (§6.3). The CS229 notes’ choice: let the source CDF be the sigmoid $g$, so $p_s = g'$ (the logistic distribution). Differentiating with respect to $W$ gives the stochastic gradient ascent rule

where $1 - 2g(\cdot)$ is applied elementwise. The first term pushes each recovered component toward the assumed density; the second, from $\log|\det W|$, keeps $W$ from degenerating (e.g. all rows collapsing onto one direction). The code averages over all samples (batch gradient ascent) and whitens first: after whitening only a rotation remains, so convergence is much faster (it also converges without whitening, just slowly — the conditioning story of Lecture 1). In practice the natural gradient (multiply the gradient by $W^\top W$ on the right, which removes the inverse; Amari, 1998) or FastICA (Hyvärinen & Oja, 2000) is more common.

6.5 The source prior matters: super- vs sub-Gaussian

演示 3，三种情形，都在白化后从 $W = I$ 出发跑 3000 步：

• (a) 成功：$|\text{corr}|$ 矩阵是排列矩阵；总变换 $WW_{\text{white}}A$ 每行恰好一个 ±1、其余接近 0 —— “排列 × 缩放”，正是 §6.2 说的不可消除的部分。
• (b) 失败，而且失败得很”有道理”：Laplace 被分出来了，正弦和锯齿波停在 $|\text{corr}| \approx 1/\sqrt2$ —— 正好是演示 2 预言的 45° 混合。这不是没收敛：在 logistic 模型下，(b) 这个错误解的似然（−4.267）高于 (c) 那个正确解（−4.725）。模型假设错了，MLE 就”正确地”找到了错误的答案 —— 和 §2.4 里 k-means 的错误答案 $J$ 更低是同一种现象。
• 为什么？ 看演示 2 的 0° 那一列：Laplace（−2.746）> 高斯（−2.858）> 正弦 + 锯齿波（−2.960）。方差固定（白化）之后，logistic 的 log 密度在尾部像 $-|y|$，平均 log 似然 ≈ 常数 $- c\,\mathbb{E}|y|$：分布越”尖”（越 super-Gaussian），$\mathbb{E}|y|$ 越小、似然越高。所以固定的 logistic 先验其实是在把每个分量推向”最尖”。而混合会把分布往高斯拉（中心极限定理的直觉）：Laplace 混合后变得不那么尖，似然下降；sub-Gaussian 的信号本来比高斯”更平”（单位方差下峰度 1.50、1.79，高斯是 3），混合后反而变尖，似然上升。
• (c) 修法：让每个分量的先验跟着它的峰度走（extended infomax, Lee, Girolami & Sejnowski, 1999）。score function 取 $\psi(y) = -y - k_j\tanh y$：$k_j = +1$ 对应一个比高斯更尖的密度，$k_j = -1$ 对应双峰的 $\tfrac12[\mathcal{N}(1,1) + \mathcal{N}(-1,1)]$，$k_j$ 每步由一个稳定性判据自动选择。三个源全部分离。

Why does CS229’s sigmoid work for the cocktail party? Because speech is super-Gaussian — near zero most of the time, occasionally loud. The notes call the sigmoid a reasonable default for most problems; that holds for audio and for filter responses of natural images, but not for regular signals such as sines. It is the Lecture 1 thread once more: choosing a prior / noise model = choosing a loss. It need not be exactly right, but its qualitative shape (here, the direction of the kurtosis) must be.

6.6 Gaussian sources in practice

演示 4 混合 1 个 Laplace 源和 2 个高斯源：

• Laplace 源每次都被干净地分离（$|\text{corr}| \ge 0.997$）：一个非高斯源的方向是确定的。
• 同一份数据、5 个随机初始化：3000 步时两个高斯分量各不相同（$|\text{corr}|$ 0.972–0.990），10000 步时都漂到了同一个解（0.983–0.985），全部 10 个结果的 $\ell$ 只差 $8\times10^{-5}$。
• 这个”共同的解”从哪来？换 6 份独立抽样的数据（同一个 $A$），它和真实高斯源的 $|\text{corr}|$ 在 0.85 到 1.00 之间随机变化。5000 个高斯样本本身不是严格高斯的（样本峰度 2.89–3.09），有限样本给了一点虚假的偏好，方向由抽样噪声决定，和源本身无关。在解附近旋转，高斯平面里的似然只变 0.0076，把 Laplace 分量拿去混合则变 0.031。

So this is what “unidentifiable” looks like in a finite sample: an almost flat ridge with small random bumps, along which gradient ascent drifts slowly. What ICA can do is separate every non-Gaussian source and hand back the Gaussian part as one undivided subspace.

7. Self-supervised learning & foundation models (briefly)

本节没有配套代码：玩具规模的模拟说明不了多少问题，相关的实践在同目录的 CS224N 和 CS336 里。

7.1 From “find the structure” to “learn a representation”

§2–§6 的方法都是我们指定结构（簇、低秩、独立源），算法去找参数。CS229 main notes 最后一章讲的是另一种思路：用海量无标签数据训练一个大模型，学出一个通用的表示 $\phi_\theta(x)$，再拿去适配各种下游任务。两个阶段：

• Pretraining：在无标签数据上优化一个 self-supervised loss —— “标签”由数据自己生成（下一个 token、被遮住的部分、同一张图的另一个增强版本……）。
• Adaptation：linear probe（冻结 $\phi_\theta$，只在上面训练一个线性模型 —— 第 1 讲的 GLM 头）；finetuning（所有参数一起更新，通常只需要少量标注）；对大语言模型还可以 zero-shot 或 in-context learning（不更新参数，把任务描述和例子写进 prompt）。

“Foundation model”（Bommasani et al., 2021）指的就是这种”一次预训练、到处适配”的模型。它改变了标注数据的角色：标签不再用来”从零学特征”，而是用来”把已有的表示对准任务”。

7.2 Two families of self-supervised losses

Contrastive learning (SimCLR, Chen et al., 2020; CLIP, Radford et al., 2021): two random augmentations of the same image form a positive pair, and the other examples in the batch are negatives. The loss (InfoNCE) is a softmax cross-entropy that asks each example to “pick out its other half” within the batch:

This is just Lecture 1’s softmax regression, with the “classes” being the examples in the batch. A good representation pulls together what survives augmentation and pushes different examples apart.

Generative / predictive pretraining：BERT 的 masked language modeling；GPT 的 next-token prediction $\max_\theta \sum_t \log p_\theta(x_t \mid x_{<t})$。后者就是对 $p(x)$ 做最大似然 —— 用链式法则把 $p(x)$ 拆成条件概率，每一个都是一个 softmax 分类器。从这个角度看，LLM 预训练是本讲”密度估计”的极端规模版本。Transformer、tokenizer、scaling law 和大规模训练的细节见 CS224N、CS336。

7.3 The VAE: EM with a learned, approximate E-step

§3.3 的分解 $\log p(x;\theta) = \text{ELBO}(x; Q, \theta) + \text{KL}(Q \Vert p(z \mid x;\theta))$ 对任何隐变量模型都成立。EM 依赖两件事：E 步能算出精确后验，M 步能精确最大化。当 $p_\theta(x \mid z)$ 是一个神经网络（decoder）时，两件都做不到。VAE（Kingma & Welling, 2014）的做法：

ELBO 可以写成”重构 + 正则”：

第一项是 autoencoder 的重构项（高斯 decoder 时就是平方误差），第二项让编码不偏离先验 $p(z) = \mathcal{N}(0, I)$。把 encoder、decoder 都换成线性、$q$ 取精确后验，就回到了 probabilistic PCA / FA。所以 VAE 是这一讲的自然终点：EM → variational EM → amortized variational inference。扩散模型的训练目标也可以写成一个 ELBO。

8. 小结 · Summary

Five takeaways:

1. Latent-variable MLE is hard because of the log-sum; EM maximizes a lower bound instead. $\log p(x) = \text{ELBO} + \text{KL}$: the E-step closes the gap, the M-step raises the bound, so $\ell$ never decreases. The same identity is the training objective of VAEs and diffusion models.
2. Alternating exact optimization is one pattern, seen twice: k-means (coordinate descent on $J$) and EM (coordinate ascent on the ELBO) are the same algorithm with hard vs soft assignments; k-means is the $\sigma \to 0$ limit.
3. The objective can prefer the wrong answer. k-means on elongated clusters, sigmoid-ICA on sub-Gaussian sources: no number of restarts fixes a misspecified model. Check the assumptions, not just the optimizer.
4. MLE can be ill-posed — collapsing GMM components, singular covariances when $d > n$, Heywood cases, and (from Lecture 1) separable logistic regression. Priors and structure (e.g. low rank + diagonal) fix them.
5. Second-order statistics determine a subspace, not a basis. PCA, whitening and FA identify things only up to rotation; pinning down the rotation needs non-Gaussianity (ICA) or extra assumptions.

9. 常见误区 · Common pitfalls

• “k-means converged, so it found the clusters.” It found a local optimum of $J$ — and even the global optimum can be the wrong partition when clusters are not round and equal-sized (§2.4).
• “Pick the k with the lowest training objective.” Distortion, reconstruction error (even on held-out data) and training likelihood all improve with model size. Use held-out likelihood, BIC, or the downstream task.
• “Higher GMM likelihood is always better.” Not when it comes from a collapsing component (§3.6). Floor the covariances or use a prior.
• “EM finds the MLE.” It finds a stationary point and only guarantees monotone improvement. Restart, and initialize sensibly (k-means++, or a k-means solution).
• “Always standardize before PCA” — or “never”. Standardize when units differ; don’t when variances are comparable and meaningful (§5.4).
• “The principal components are the underlying factors.” PCA (like FA) identifies a subspace; the axes are just “ordered by variance”. Recovering independent causes is ICA’s job — and only for non-Gaussian sources.
• “The ICA source prior is a detail.” Its kurtosis direction decides whether ICA separates or mixes (§6.5).

10. 练习 · Exercises

练习 1（ELBO 的分解）. 证明对任意分布 $Q(z)$，$\log p(x;\theta) = \text{ELBO}(x;Q,\theta) + \text{KL}\big(Q \Vert p(z\mid x;\theta)\big)$。由此说明：(a) E 步为什么取后验；(b) 如果 M 步只做一步梯度上升（generalized EM），$\ell$ 为什么仍然单调不减。

$\text{ELBO} = \sum_z Q(z)\log\frac{p(x,z)}{Q(z)} = \sum_z Q(z)\log\frac{p(z\mid x)\,p(x)}{Q(z)} = \log p(x) - \sum_z Q(z)\log\frac{Q(z)}{p(z\mid x)}$，最后一项就是 KL。(a) $\log p(x;\theta_t)$ 和 $Q$ 无关，所以关于 $Q$ 最大化 ELBO 等价于最小化 KL，最小值 0 在 $Q = p(z \mid x;\theta_t)$ 取到。(b) 链条 $\ell(\theta_{t+1}) \ge \text{ELBO}(Q_t,\theta_{t+1}) \ge \text{ELBO}(Q_t,\theta_t) = \ell(\theta_t)$ 里，第二个不等号只需要”M 步不让 ELBO 下降”，不需要真的最大化；步长足够小的一步梯度上升就满足。

练习 2（k-means = hard EM）. 考虑权重相等、$\Sigma_j = \sigma^2 I$ 固定的 GMM。写出 E 步的 $w_j^{(i)}$，证明 $\sigma \to 0$ 时它趋于 $1\{j = \arg\min_k\lVert x^{(i)} - \mu_k\rVert\}$（假设没有平局），并说明此时 M 步就是 k-means 的 update step。再说明 ELBO 乘以 $2\sigma^2$、去掉常数后，在这个极限下变成 $-J(c,\mu)$。

$w_j^{(i)} = \exp(-\lVert x^{(i)}-\mu_j\rVert^2/2\sigma^2)\big/\sum_k \exp(-\lVert x^{(i)}-\mu_k\rVert^2/2\sigma^2)$。分子分母同除以最近中心那一项，其余每项变成 $\exp(-\Delta_k/2\sigma^2)$，$\Delta_k > 0$，$\sigma\to0$ 时趋于 0。M 步 $\mu_j = \sum_i w_j^{(i)}x^{(i)}/\sum_i w_j^{(i)}$ 变成簇内平均。ELBO $= \sum_i\sum_j w_j^{(i)}\big[-\lVert x^{(i)}-\mu_j\rVert^2/2\sigma^2 - \log w_j^{(i)}\big] + \text{const}$；硬分配时熵项为 0，乘以 $2\sigma^2$ 得 $-\sum_i\lVert x^{(i)}-\mu_{c^{(i)}}\rVert^2 = -J$。所以 k-means 的两个半步就是 hard EM 在这个 ELBO 上的坐标上升（§3.5 的表格：$\sigma \le 0.02$ 时两者给出同一个解）。

练习 3（塌缩与先验）. 一维、两个成分的 GMM，把 $\mu_1$ 放在 $x^{(1)}$ 上，令 $\sigma_1 \to 0$，其他参数固定且 $\phi_1, \phi_2 > 0$。(a) 证明 $\ell \to +\infty$ 并给出增长速度。(b) 加上惩罚 $-\frac{\lambda}{2}\sigma_1^{-2}$ 后，目标函数在 $\sigma_1 \to 0$ 时的极限是什么？(c) 为什么”拥有”足够多、位置一般的点的成分不会塌缩？

(a) $\log p(x^{(1)}) \ge \log\big(\phi_1\,\mathcal{N}(x^{(1)};x^{(1)},\sigma_1^2)\big) = \log\phi_1 - \tfrac12\log(2\pi) - \log\sigma_1 \to \infty$；其余每个点 $\log p(x^{(i)}) \ge \log\big(\phi_2\,\mathcal{N}(x^{(i)};\mu_2,\sigma_2^2)\big)$ 有下界。所以 $\ell$ 以 $\log(1/\sigma_1)$ 的速度增长（$d$ 维时是 $d\log(1/\sigma)$，即 §3.6 里每十倍 4.61）。(b) 惩罚以 $\sigma_1^{-2}$ 的速度趋于 $-\infty$，压过 $\log(1/\sigma_1)$，极限是 $-\infty$：最优点不再在边界上。(c) 若成分 $k$ 的有效点多于 $d$ 个、且不全在一个超平面上，加权样本协方差 $S_k$ 正定，M 步的 $\Sigma_k = S_k$ 不会趋于奇异。塌缩需要成分只”拥有”不超过 $d$ 个点（它们张成至多 $d-1$ 维的仿射子空间）；§3.6 的例子是 $d = 2$ 时 2 个点塌缩到一条直线。

练习 4（PCA 的两个定义）. 设数据已中心化，$\Sigma = \frac1n\sum_i x^{(i)}x^{(i)\top}$ 的特征值 $\lambda_1 \ge \dots \ge \lambda_d$，$V_k$ 是前 $k$ 个特征向量。证明 $\frac1n\sum_i\big\lVert x^{(i)} - V_kV_k^\top x^{(i)}\big\rVert^2 = \sum_{j>k}\lambda_j$，并说明为什么同一个量在测试集上也随 $k$ 单调不增。

$V_kV_k^\top$ 是正交投影，所以 $\lVert x - V_kV_k^\top x\rVert^2 = \lVert x\rVert^2 - \lVert V_k^\top x\rVert^2$。对 $i$ 平均：$\frac1n\sum_i\lVert x^{(i)}\rVert^2 = \text{tr}\,\Sigma = \sum_j\lambda_j$，$\frac1n\sum_i\lVert V_k^\top x^{(i)}\rVert^2 = \text{tr}(V_k^\top\Sigma V_k) = \sum_{j\le k}\lambda_j$，相减即得。对任何一个点（包括测试点），子空间嵌套 $\text{span}(V_k) \subset \text{span}(V_{k+1})$，投影到更大的子空间，残差只会更小。所以测试误差也单调不增，没有 U 形，不能用来选 $k$（§5.1 的表格）。

练习 5（为什么高斯源不行）. (a) 设 $s \sim \mathcal{N}(0, I)$，$x = As$。证明对任意正交矩阵 $R$，用 $A' = AR$ 生成的数据与 $x$ 同分布。(b) 白化后 $\tilde x = Ms$。若源独立、单位方差，证明 $M$ 是正交矩阵，因此 ICA 只需在正交矩阵里搜索。(c) 恰好一个源是高斯时，为什么其余源仍可恢复？两个高斯源时会怎样？

(a) $x' = A's = ARs$，而 $Rs \sim \mathcal{N}(0, RR^\top) = \mathcal{N}(0, I)$，所以 $x' \overset{d}{=} As = x$，任何数据量都区分不了 $A$ 和 $AR$。(b) $\text{Cov}(\tilde x) = M\,\text{Cov}(s)\,M^\top = MM^\top = I$，所以 $M$ 正交，$W = M^\top$ 也正交，只剩 $d(d-1)/2$ 个自由度。(c) 非高斯源的方向由高阶统计量确定（Comon 定理：最多一个高斯源时，独立分量在排列和缩放意义下唯一）；只有一个高斯源时，它占据的是”其余方向的正交补”，一维里的”旋转”只剩符号。两个高斯源时，它们张成的二维子空间里的旋转无法确定 —— §6.6 的实验里，这个旋转随初始化、迭代步数和抽样噪声漂移。

练习 6（sigmoid 先验与峰度）. (a) 单位方差下，Laplace、高斯、均匀分布的 $\mathbb{E}|y|$ 分别是 $1/\sqrt2 \approx 0.707$、$\sqrt{2/\pi} \approx 0.798$、$\sqrt3/2 \approx 0.866$。用”logistic 的 log 密度在尾部像 $-|y|$“解释：为什么在固定的 logistic 先验下，两个 Laplace 源的 45° 混合似然更低，而两个均匀分布（锯齿波）源的 45° 混合似然更高？(b) 如果事先知道所有源都是 sub-Gaussian，应该怎样修改 CS229 的更新规则？

(a) 白化固定了方差，平均 log 似然 ≈ 常数 $- c\,\mathbb{E}|y|$，所以 $\mathbb{E}|y|$ 越小越好，即越”尖”越好。混合（中心极限定理）把分布往高斯拉：Laplace 的混合 $\mathbb{E}|y|$ 从 0.707 往 0.798 变大，似然下降；均匀分布的混合从 0.866 往 0.798 变小，似然上升。固定先验下的 MLE 把每个分量推向”最尖”，对 sub-Gaussian 源来说，最尖的恰好是混合（§6.3 的表格）。(b) 换一个 sub-Gaussian 的先验，例如双峰的 $p(s) = \tfrac12[\mathcal{N}(1,1) + \mathcal{N}(-1,1)]$，它的 score 是 $\psi(y) = -y + \tanh y$，更新规则变成 $W := W + \alpha\big(\psi(Wx^{(i)})\,x^{(i)\top} + (W^\top)^{-1}\big)$。不知道源的类型时，用 extended infomax 让每个分量自己选（§6.5 (c)）。FastICA 用 $\big(\mathbb{E}[G(y)] - \mathbb{E}[G(\nu)]\big)^2$（$\nu$ 是标准高斯）这类”偏离高斯的程度”做目标，两种方向都能处理。