Lecture 2 · 生成式学习、核方法与 SVM · Generative Learning, Kernels & SVMs

对应 Maps to：Stanford CS229 Autumn 2018（Andrew Ng）Lectures 5–7（GDA & Naive Bayes; Support Vector Machines; Kernels），以及 CS229 main notes（Andrew Ng & Tengyu Ma）Part I, Chapters 4–6。 CS229 当前学期（Summer 2026）的课表需要 Stanford 登录才能看，所以本讲以公开的讲义和 2018 年的录像为准；这部分内容多年来基本没变。 配套代码 Code：code/lecture02/（numpy；2 个工具模块 + 4 个脚本。04_svm_smo.py 约 10 秒，其余都在 2 秒以内；复用第 1 讲的 linear_models.py 和 datasets.py） 预计阅读时间：2.5–3 小时（含运行代码）。这是一份复习讲义：假设你见过 GDA、Naive Bayes、核函数和 SVM，重点放在”为什么”和”它们之间的联系”上。

0. 本讲导航 · How to read this lecture

0.1 Learning goals

After this lecture you should be able to:

1. 说清楚 discriminative 和 generative 两条路各自对什么建模、各自付出什么代价，并用 GDA vs logistic regression 的例子解释”更省数据”和”更稳健”为什么不可兼得。
2. Derive in one line why GDA’s posterior is exactly a logistic function — the “third origin of the sigmoid” promised in Lecture 1 §3.2 — and why the converse is false.
3. 解释 Naive Bayes 的条件独立假设、两种 event model、Laplace smoothing；说清楚”假设错了为什么还能分类”以及”错在哪里”（过度自信的概率）。
4. Explain the kernel trick as a consequence of two facts — the algorithm only touches inner products and the solution lies in the span of the data — and decide when a function is a valid kernel.
5. 从”最大化几何间隔”出发得到 SVM，读懂对偶问题和 KKT 条件给出的结论（support vectors、只用内积、soft margin = hinge + L2、SMO 为什么一次改两个变量），并兑现第 1 讲 §3.5 的承诺：可分数据上 GD 的方向收敛到 max-margin 方向。

0.2 默认你已经会的 / 本讲会省略的

0.3 本讲地图 Map

§1  Discriminative vs generative：对 p(y|x) 建模，还是对 p(x|y)p(y) 建模
§2  GDA：封闭解 / 后验恰好是逻辑回归 / 更省数据 vs 更稳健                 ← 代码 01
§3  Naive Bayes：条件独立 / 两种 event model / Laplace smoothing / 过度自信  ← 代码 02
§4  Kernel methods：θ 在数据张成的空间里 / kernel trick / 合法的核 / kernel ridge  ← 代码 03
§5  SVM：间隔 / 对偶与 KKT / support vectors / soft margin / SMO / GD 的隐式偏置  ← 代码 04
§6–10 小结、常见误区、练习、延伸阅读、下一讲预告

0.4 Notation（新增的符号）

第 1 讲把截距当作 $x$ 的一个分量（$x_0 = 1$）；SVM 按 CS229 讲义的写法把 $b$ 单独拿出来。这个看似无关紧要的差别在 §5.7 会变得重要。

0.5 关于数字

讲义里所有实验数字都来自配套代码的实际输出（固定随机种子，同一台机器上重复运行逐字相同）。数据是合成的：我们知道”真相”，所以能检查算法有没有学对；数字本身不代表任何真实系统。和预想不一样的结果都如实写出来了，并记录在代码 README 的”关于实验结果”里。

1. Discriminative vs generative · 判别式与生成式

1.1 Two ways to build a classifier

第 1 讲的模型都直接对 $p(y \mid x)$ 建模：给定 $x$，$y$ 是什么分布。这叫 discriminative（判别式）学习。另一条路是对数据是怎么产生的建模：先抽一个类别 $y \sim p(y)$，再从这一类的分布里抽 $x \sim p(x \mid y)$。这叫 generative（生成式）学习。预测时用 Bayes 公式反推：

分类只需要 $\arg\max_y p(x \mid y)\,p(y)$，分母 $p(x)$ 对所有类别相同，可以不算。

Andrew Ng 的比喻：要区分大象和狗，判别式方法找一条把两类分开的边界；生成式方法分别画一张”大象长什么样”和一张”狗长什么样”，来了新动物就看它更像哪一张。

1.2 What each side pays for

Vapnik’s often-quoted principle says, roughly: don’t solve a harder, more general problem as an intermediate step toward the one you care about. Estimating $p(x \mid y)$ is harder than estimating a boundary ($x$ is high-dimensional), which is why discriminative methods usually win when data are plentiful. 但”更难的问题”也带来了更多的结构：假设对的时候，它把这些结构变成了统计效率。§2.4 用实验把这句话变成数字。

2026 年的视角：今天的”generative model”（LLM、diffusion）建模的是 $p(x)$ 本身，目的是生成而不是分类；但”对数据的生成过程建模”这个思想和本讲的 GDA / Naive Bayes 一脉相承。

2. Gaussian discriminant analysis · 高斯判别分析

2.1 The model

关键的设计决定是两类共享同一个协方差 $\Sigma$：两团形状、朝向都一样，只有中心不同。（统计学里这叫 linear discriminant analysis，LDA；两类各用自己的 $\Sigma_c$ 叫 QDA。）

2.2 MLE in closed form

最大化联合似然 $\sum_i \log p(x^{(i)}, y^{(i)})$，各参数解耦，答案就是”数一数、求平均”：

$\Sigma$ 是 pooled covariance：每个点减去自己那一类的均值。不需要任何迭代 —— 对比逻辑回归要 GD 或 Newton。代码 01_gda_vs_logistic.py 演示 1（$d = 2$，每类 200 个点，真实 $\Delta = 2$）：估计的 $\mu_1 = [1.756, 1.669]$（真值 $[1.732, 1.732]$），$\hat\Sigma = \begin{bmatrix}0.919 & 0.398\\0.398 & 0.946\end{bmatrix}$（真值对角 1、非对角 0.5）。

2.3 The posterior is exactly logistic — the third origin of the sigmoid

Take the log-odds and plug in the two Gaussian densities. Because both classes share $\Sigma$, the quadratic terms $-\frac12 x^\top\Sigma^{-1}x$ cancel, and what remains is linear in $x$:

The code checks this literally: gda_predict_proba computes the posterior from two Gaussian densities and Bayes’ rule — no sigmoid anywhere — and on 10,000 new points it agrees with $\sigma(\theta_0 + \theta^\top x)$ to $7.8\times10^{-16}$. This completes Lecture 1 §3.2’s list: the sigmoid is (1) a convenient squashing function, (2) the canonical response of the Bernoulli GLM, and (3) the exact posterior of Gaussian class-conditionals with shared covariance.

The converse is false. Demo 1(b) replaces the Gaussians with independent Poissons, $x_j \mid y \sim \text{Poisson}(\lambda_{y,j})$. The posterior is again exactly logistic, with $\theta_j = \log(\lambda_{1j}/\lambda_{0j})$ (max difference $1.6\times10^{-15}$). In general, any class-conditionals from the same exponential family that differ only in the natural parameter give a logistic posterior. So:

逻辑回归的假设严格更弱：它不在乎 $p(x \mid y)$ 长什么样，只要 log-odds 是线性的。

同一个模型族，不同的估计量。 GDA 和逻辑回归的 hypothesis class 一模一样（都是 $\sigma(\theta_0 + \theta^\top x)$），但它们挑 $\theta$ 的方式不同：GDA 最大化联合似然 $p(x, y)$，逻辑回归最大化条件似然 $p(y \mid x)$。演示 1 在同一份数据上：

2.4 GDA vs logistic: efficiency vs robustness

Which estimator is better? It depends on whether the Gaussian assumption is true. 演示 2、3 用学习曲线回答（$d = 10$，$\Sigma_{jk} = 0.5^{|j-k|}$，$\Delta = 2.5$；每个样本量重复 200 次；错误率是线性分类规则在总体上的精确错误率 —— 用 $\Phi(\cdot)$ 的封闭式算，不受测试集噪声影响）。

假设成立时（Bayes 最优 0.1056）：

“多犯的错”= 错误率 − Bayes 最优。从每类 15 个样本起 GDA 都更好，每类 ≥ 25 个时逻辑回归多犯的错是 GDA 的 1.2–1.3 倍，而且这个比值不随 $n$ 趋于 1 —— 这就是 CS229 讲义说的”模型正确时 GDA 是 asymptotically efficient 的”：它用上了”$x$ 是高斯的”这条额外信息。（Efron 1975 专门算过逻辑回归相对于 normal discrimination 的渐近效率：两类分得越开，逻辑回归吃亏越多。）

But look at the first column: with only 10 points per class (20 points in $d = 10$) GDA is worse. It has to estimate the 55 entries of $\Sigma$; with so few samples $\hat\Sigma$ is poor and $\hat\Sigma^{-1}$ amplifies the error. The word asymptotically matters — at tiny sample sizes, fewer parameters (or regularization, see Exercise 2 in the code README) matter more. This contradicts the simple story “GDA is always more data-efficient”, and we report it as it came out.

假设不成立时。 演示 3 让正类有一条”单侧长尾”：10% 的正类样本被额外推远 $5(\mu_1 - \mu_0)$，落在离边界很远、毫无争议的一侧 —— 就是第 1 讲 §3.1 那些”特别大的肿瘤”的高维版本。$p(x \mid y = 1)$ 不再是高斯，但边界附近的后验几乎还是 sigmoid。Bayes 最优（Monte Carlo 估计）= 0.1001：

Logistic regression wins at every sample size, and GDA’s gap does not shrink with $n$: it stalls at 0.1362, 0.036 above Bayes, while logistic regression converges to 0.1002. Split by class (at $10^5$ per class), GDA errs 0.0297 on negatives but 0.2426 on positives: the tail drags $\hat\mu_1$ away by $0.51\lVert\mu_1 - \mu_0\rVert$ ($= 10\% \times 5$) and stretches $\hat\Sigma$ along the separating direction, pushing the boundary into the positive class. Logistic loss goes to 0 at large margins, so those far, already-correct points contribute almost no gradient — the same mechanism as the huge tumors of Lecture 1.

两张表合起来就是 Ng & Jordan (2002) 的结论：generative 模型更快地接近它自己的渐近错误率；discriminative 模型的渐近错误率更低或相同。经验法则：

• 数据少、对 $p(x \mid y)$ 的形状有把握 → 生成式（GDA、Naive Bayes）；
• 数据多、假设可疑 → 判别式（逻辑回归、SVM、神经网络）。
• 差别的本质是 bias–variance（第 3 讲）：更强的假设 = 更小的方差 + 可能的偏差。

3. Naive Bayes · 朴素贝叶斯

3.1 Why an assumption is unavoidable

垃圾邮件分类：$x \in \{0, 1\}^V$ 表示词表里每个词出现没有。直接对 $p(x \mid y)$ 建模，一个一般的分布需要 $2^V - 1$ 个参数 —— $V = 50000$ 时完全不可能。必须做一个强假设。Naive Bayes 的假设是：给定类别，各个词相互独立（conditional independence）：

参数从 $2^V$ 个降到 $2V + 1$ 个。注意右边：log-odds 是每个词的证据之和 —— 又是一个线性模型（在 $x$ 的指示特征上），所以 Naive Bayes 也是 §2.3 那种”生成式模型的后验是逻辑函数”的例子。这个假设显然是错的（“click” 和 “here” 显然不独立），§3.4 讨论为什么它还能用。

3.2 Two event models

代码 02_naive_bayes.py 构造了一个合成语料：$V = 300$（20 个常用词、15 个垃圾邮件词、15 个正常邮件词、250 个按 Zipf 定律递减的长尾词），每封邮件平均 20 个词，逐词独立地从本类的词分布里抽 —— multinomial NB 的假设恰好成立，用真实分布做 Bayes 最优分类的准确率是 0.9535。演示 1 的学习曲线（每个大小重复 10 次；NB 用 $\alpha = 1$，逻辑回归用词频做特征、$\lambda = 10^{-3}$）：

• NB vs logistic. With little data NB is far better (25 emails per class: 0.867 vs 0.745); the gap shrinks to 0.006 at 1000 per class. NB’s “training” is counting, with each word estimated independently; logistic regression optimizes 301 parameters jointly and has high variance when data are scarce. This is the other half of Ng & Jordan (2002). But our corpus is generated exactly by NB’s model, so logistic regression never overtakes it — the crossover Ng & Jordan saw on real datasets (the discriminative model winning once data are plentiful) cannot happen in this toy world.
• Bernoulli vs multinomial. 本以为 multinomial 会明显更好，实际上数据多时几乎一样（0.9471 vs 0.9475），训练集极小时 Bernoulli 反而略好（我们没有深究原因）。邮件很短，关键词很少重复，“出现几次”和”出现没有”差别不大。文献里的经验（McCallum & Nigam 1998）是文档长、词表大时 multinomial 占优。

3.3 Laplace smoothing: why one zero count kills a prediction

MLE 把”训练集里没见过”估计成”概率恰好为 0”。演示 2 用每类 25 封邮件（约 1000 个词）训练：300 个词里有 218 个在 ham 训练集中从未出现、207 个在 spam 中从未出现。测试邮件里只要出现一个”spam 训练集里没见过”的词，$\log p(x \mid \text{spam}) = -\infty$ —— 无论其他词多么像垃圾邮件。

Laplace smoothing 给每个计数加 $\alpha$（$\alpha = 1$ 就是”每个词先假装见过一次”）：

• $\alpha = 0$: 3129 of 4000 test emails have probability exactly 0 under both classes (for each class they contain at least one word that class never showed in training), so the posterior is 0/0 and no decision is possible (counted as wrong). With a large vocabulary and a small training set this is almost guaranteed.
• Any $\alpha > 0$, even 0.001, brings accuracy to 0.87. Accuracy is fairly insensitive to $\alpha$; log loss is very sensitive — a tiny $\alpha$ lets unseen words push probabilities to the extremes.
• $\alpha$ 太大也不行：$\alpha = 10$ 时每个词”假装见过 10 次”，伪计数（$10 \times 300$）远多于每类真实的约 500 个词，信号被抹平。同一个 $\alpha$ 对两种 event model 的强度不同（Bernoulli 的伪计数相对于 25 封邮件，multinomial 的相对于约 500 个词），所以大 $\alpha$ 时 Bernoulli 掉得慢。

一个构造的例子把”一票否决”看得最清楚：取 8 个真正像垃圾邮件的词（deal winner offer prize free 和 3 个长尾词），再加一个 spam 训练集里从未出现、ham 里出现过 5 次的词 “lunch”：

log 概率是逐词相加的，一个 $-\infty$ 压倒一切。$\alpha > 0$ 时 “lunch” 只是一条普通的、有限的反向证据。

Why “+α” is principled, not a hack. Laplace introduced add-one smoothing for the sunrise problem: having seen the sun rise $n$ times, the probability it rises tomorrow is $(n+1)/(n+2)$, not 1. In Bayesian terms, $\frac{\text{count} + \alpha}{N + \alpha V}$ is the posterior mean under a symmetric Dirichlet($\alpha$) prior on the word distribution (equivalently, the MAP estimate under Dirichlet($\alpha + 1$)). So $\alpha$ is a regularization strength expressed as pseudo-counts — the same idea as ridge = Gaussian prior in Lecture 3. 在这份数据上 log loss 最好的是 $\alpha = 0.3$，不是 1：它是一个要调的超参数。

3.4 Why the wrong assumption still classifies well — and what it breaks

The independence assumption is wrong for every real text corpus. Yet NB is a famously strong baseline. The reason is that classification only needs the sign of the log-odds, while probabilities need its size. Correlated words make NB count the same evidence several times, which inflates the log-odds but often doesn’t flip its sign (Domingos & Pazzani 1997 showed NB can be optimal under 0-1 loss even when independence is badly violated).

演示 3 把这件事做成一个可控的实验：先在”NB 假设恰好成立”的语料上训练（$k = 1$），再把每个词换成 $k$ 个总是一起出现的不同的词（像 “click here now”），词表变成 $300k$ 个。NB 以为它们相互独立，于是同一份证据被数了 $k$ 遍（每类 250 封训练、2000 封测试）：

（ECE = expected calibration error：按置信度分箱，比较”说有多确定”和”实际有多准”。）

• Accuracy doesn’t move in the 4th decimal. After duplication each copy’s estimated probability is exactly $1/k$ of the original and every log-odds is exactly $k$ times the original (max difference $4.5\times10^{-13}$, Exercise 4). Same sign, same decisions.
• Probabilities fall apart. At $k = 1$ NB is well calibrated; at $k = 8$, 93.5% of predictions claim to be more than 99.9% sure, yet only 96% of those are right, and the log loss is 4.4 times larger.
• Logistic regression fits $p(y \mid x)$ directly, so $k$ co-occurring features just share one weight instead of being counted $k$ times: its log loss only moves from 0.193 to 0.218. It isn’t untouched — the fixed L2 penalty spread over $k$ copies is effectively $k$ times weaker, so variance grows and accuracy drops from 0.9275 to 0.9135 — but nothing like NB.

实践结论：NB 是很好的排序 / 分类器（垃圾邮件过滤、文本分类的强 baseline；NB 特征 + SVM 的 NBSVM 至今仍是情感分类的经典 baseline，Wang & Manning 2012），但别把它的输出直接当概率用。需要概率时做一次校准（Platt scaling、isotonic regression），或者换判别式模型。

4. Kernel methods · 核方法

4.1 From features to inner products

第 1 讲 §2.5 说过：“线性模型”只要求对 $\theta$ 线性，可以把 $x$ 换成任意特征 $\phi(x)$。问题是好的特征往往非常多：$d$ 维输入的所有次数 $\le p$ 的单项式有 $\binom{d+p}{p}$ 个（代码 03_kernels.py 演示 1）：

MNIST 上的 4 次多项式特征，一个样本的特征向量就要 128 GB。在这么多特征上跑 LMS（每步 $O(nD)$）是不可能的。

4.2 The kernel trick: θ stays in the span of the data

Run batch GD (LMS) in feature space from $\theta = 0$:

Every update adds a linear combination of the $\phi(x^{(i)})$, so by induction $\theta = \sum_i \beta_i\,\phi(x^{(i)})$ at all times. Substitute this in and $\phi$ only ever appears inside inner products:

If $K(x, z)$ can be computed without forming $\phi$, we never pay for the dimension $D$. For the polynomial kernel this is immediate: $(x^\top z + c)^2$ costs one $d$-dimensional inner product, yet equals $\phi(x)^\top\phi(z)$ for a $(d^2 + d + 1)$-dimensional $\phi$ (demo 1(a): max difference $7.1\times10^{-15}$). Demo 1(c) runs both versions of GD for 500 steps on 13-dimensional features: explicit $\theta$ and $\sum_i\beta_i\phi(x^{(i)})$ agree to $1.0\times10^{-15}$, predictions to $4.0\times10^{-15}$.

这就是 kernel trick 的全部前提，两个条件缺一不可：

1. 算法只通过内积接触数据（LMS、ridge、SVM、PCA、k-means 都可以写成这样）；
2. 解在数据张成的空间里（representer theorem：对”损失 + $\lVert\theta\rVert^2$ 正则化”这一大类问题都成立）。

第 1 讲 §2.3 已经见过第二条的影子：$d > n$ 时从 0 出发的 GD 永远不会长出 $X^\top$ 列空间之外的分量，所以收敛到最小范数解。

4.3 RBF: an infinite-dimensional feature map

The Gaussian (RBF) kernel $K(x, z) = \exp\big(-\lVert x - z\rVert^2/(2\sigma^2)\big)$ corresponds to an infinite-dimensional $\phi$. In 1-D, Taylor-expand $e^{xz/\sigma^2}$:

演示 1(b) 把这个无穷级数截断到前 $K$ 维（$\sigma = 1$），看近似误差：

误差随 $K$ 下降得极快，但输入范围越大需要的维数越多 —— 没有任何有限的 $K$ 对所有输入都够用。核函数用 $O(d)$ 的代价”算完了”这个无穷级数。直觉上，RBF 是一个”相似度”：$x = z$ 时为 1，离得越远越接近 0，$\sigma$ 决定”多远算远”。

4.4 Which functions are valid kernels?

Not every similarity function is an inner product in some feature space. Mercer’s theorem (finite-sample form, as in the CS229 notes): $K$ is a valid kernel if and only if for every finite set of points the Gram matrix $K_{ij} = K(x^{(i)}, x^{(j)})$ is symmetric positive semidefinite. 必要性一行就能看出来：$v^\top K v = \lVert\sum_i v_i\phi(x^{(i)})\rVert^2 \ge 0$。

演示 2 在 60 个随机二维点上算 Gram 矩阵的特征值：

• 合法的核做加法、乘法、正数缩放、$f(x)f(z)K(x,z)$ 仍然合法 —— 这是构造新核的标准工具（和 = 特征拼接，积 = 特征的张量积）。
• “sigmoid 核” $\tanh(a\,x^\top z + b)$ 在早期 SVM 文献里很流行（像一层神经网络），但这里 60 个点中有 30 个 $K(x, x) < 0$ —— 如果 $K(x, x) = \lVert\phi(x)\rVert^2$，这就是”长度的平方是负数”。
• box 核非常符合”相似度”的直觉（对称、自己和自己最像），却不合法：取最小特征值的特征向量 $v$，$v^\top K v = -4.26$，相当于 $\lVert\sum_i v_i\phi(x^{(i)})\rVert^2 < 0$。（Bochner 定理的直觉：平移不变的核合法 ⇔ 它的 Fourier 变换非负；盒子的 Fourier 变换是 sinc，有负的部分；高斯的还是高斯。）
• 不合法的后果：SVM 的对偶问题不再凹（没有唯一最优，SMO 的曲率可能 ≤ 0）；kernel ridge 的 $K + \lambda I$ 可能奇异。一个负特征值就足以证明不合法；一组点上没有负特征值只是”没找到反例”。

4.5 Kernel ridge regression and the O(n³) vs O(D³) trade-off

Ridge in feature space, $\min_\theta \sum_i (y^{(i)} - \theta^\top\phi(x^{(i)}))^2 + \lambda\lVert\theta\rVert^2$, has two equivalent closed forms (push-through identity):

Primal solves a $D \times D$ system, dual an $n \times n$ one. Demo 3 checks they give the same predictions (difference $2.4\times10^{-14}$ with a quadratic kernel). 演示 3 的主角是第 1 讲 §2.5 的正弦数据 —— 同一份 15 个训练点（第 1 讲用多项式特征时最好的是 $k = 3$，test MSE 0.159）。RBF 核、$\lambda = 0.1$，扫带宽 $\sigma$：

“有效自由度” $= \text{tr}\big(K(K + \lambda I)^{-1}\big)$，衡量模型有多少个”有效参数”（和第 1 讲 LWR 的”有效样本数”是同一类量）。Small $\sigma$: each training point lights up only a tiny neighborhood, so the fit is exact at the training points and close to 0 elsewhere — overfitting. Large $\sigma$: all points look “equally similar” and the curve is flattened — underfitting. The best, $\sigma = 0.2$, reaches 0.133, better than the best polynomial on the same data (0.159).

固定 $\sigma = 0.2$ 扫 $\lambda$：$\lambda = 10^{-6}$ 时有效自由度 10.5，曲线穿过噪声点，test MSE 3.17；$\lambda = 0.01$、$0.1$ 时 0.143、0.133；$\lambda = 10$ 时曲线被压向 0，test MSE 0.402。RBF 核 = 无穷维特征 + ridge：表达能力无限，靠 $\sigma$ 和 $\lambda$ 两个旋钮控制复杂度（第 3 讲用交叉验证选）。

When to use the dual. $D$ small and $n$ large (ordinary linear regression) → primal. $D$ huge or infinite (high-degree polynomial, RBF) → dual. The price is the $n \times n$ Gram matrix: $O(n^2)$ memory and $O(n^3)$ time. At $n = 10^6$ the Gram matrix alone has $10^{12}$ entries (8 TB). This is the main reason kernel methods lost ground to neural networks on large datasets; the standard escapes are random Fourier features (Rahimi & Recht 2007: sample a finite $\phi$ whose inner product approximates the kernel, then solve the primal) and Nyström (low-rank approximation of $K$).

4.6 Kernels in 2026

• Gaussian processes: the kernel-ridge prediction is exactly the posterior mean of a GP with covariance $K$ and noise variance $\lambda$; the GP adds error bars. Bayesian optimization of hyperparameters is built on this.
• Neural tangent kernel (Jacot et al. 2018): an infinitely wide network trained by GD with a small learning rate behaves like kernel regression with a specific kernel. Much of deep-learning theory is phrased in kernel language.
• Attention (Lecture 1 §2.5): $\sum_i\text{softmax}(q^\top k_i)\,v_i$ is a kernel smoother with a learned, asymmetric similarity — not a Mercer kernel, which is fine because it is never used as an inner product in a feature space.

5. Support vector machines · 支持向量机

5.1 Margins: functional vs geometric

Lecture 1 §3.4 put every classification loss on the axis of the margin $t = y\,\theta^\top x$. SVMs take the margin as the objective. With $y \in \{-1, +1\}$ and $f(x) = w^\top x + b$:

几何间隔是点到超平面的带符号欧氏距离。functional margin 可以靠把 $(w, b)$ 乘以 10 任意放大，超平面却没变（代码 04_svm_smo.py 演示 1：最小 functional margin 1.000 → 10.000，几何间隔 0.9396 → 0.9396）。所以只有几何间隔是有意义的”置信度”。

5.2 The optimal margin classifier

Maximize the smallest geometric margin. Since $(w, b)$ can be rescaled freely, fix the smallest functional margin to 1; maximizing $1/\lVert w\rVert$ is then the convex QP

为什么”最大间隔”是好的归纳偏置？ 三个角度：

• Robustness. With margin $\gamma$, every training point can be perturbed by up to $\gamma$ without changing its label. If test points look like noisy versions of training points, a wider margin is safer.
• Dimension-free generalization. For data in a ball of radius $R$, the effective capacity (VC dimension) of linear classifiers with margin $\gamma$ is controlled roughly by $R^2/\gamma^2$ rather than by the dimension $d$ (Vapnik). This is the theoretical reason an SVM can work in the infinite-dimensional RBF feature space without overfitting. Lecture 1’s Novikoff bound $(R/\gamma)^2$ is the same quantity — the margin sets the difficulty.
• Uniqueness. Separable data admit infinitely many separators (where the perceptron stops depends on the example order); the maximum-margin one is unique.

演示 1 用第 1 讲感知机那份可分数据比较几个分隔面的间隔：

第 1 讲演示 4 的”间隔”（0.1759、0.3152）是在增广空间里量的（除以包括截距的 $\lVert\theta\rVert$），所以和这里的第一列不是同一个量；两种都列出来了。用增广空间里最大的间隔，Novikoff 上界从第 1 讲的 759 降到 629，仍远大于感知机实际犯错的 10 次。

5.3 Lagrange duality, KKT and support vectors (results only)

Introducing a multiplier $\alpha_i \ge 0$ per constraint, the dual problem is

and the KKT conditions give three facts we actually use:

1. $w = \sum_i\alpha_i y^{(i)}x^{(i)}$ — the solution is a combination of training points (the representer theorem again).
2. Complementary slackness: $\alpha_i\big(y^{(i)}(w^\top x^{(i)} + b) - 1\big) = 0$. Points strictly outside the margin have $\alpha_i = 0$; only points on the margin can have $\alpha_i > 0$. These are the support vectors.
3. Strong duality holds: the optimal primal and dual objectives are equal — a free convergence check.

The equality constraint $\sum_i\alpha_i y^{(i)} = 0$ comes from differentiating with respect to $b$; remember it, it explains SMO (§5.6).

演示 1 的数字：100 个点里只有 3 个 support vectors（$\alpha = 0.137, 0.429, 0.566$），它们的 functional margin 恰好都是 1，其余点最小 1.041；$\sum\alpha_i y_i = 2\times10^{-16}$；primal $\frac12\lVert w\rVert^2$ = dual $W(\alpha)$ = 0.566406。稀疏性的直接证明：扔掉其余 97 个点、只用 3 个 support vectors 重新训练，$(w, b)$ 的最大差是 $3.6\times10^{-11}$。预测 $f(x) = \sum_i\alpha_i y^{(i)}\langle x^{(i)}, x\rangle + b$ 也只需要对 support vectors 求和。

5.4 Kernel SVM

The dual and the prediction touch the data only through $\langle x^{(i)}, x^{(j)}\rangle$ — condition 1 of §4.2. Replace it by $K(x^{(i)}, x^{(j)})$ and the same QP, the same solver and the same code learn a nonlinear boundary. 演示 3：同心圆（内圈负类、外圈正类，训练 200、测试 2000，$C = 1$）：

• No line separates concentric circles (0.5705 ≈ guessing). The degree-2 polynomial kernel’s features include $x_1^2$ and $x_2^2$, in which “radius squared = constant” is a hyperplane: 0.978 with only 19 support vectors. The model and the optimizer did not change by a single character; only $K$ did.
• RBF 的 $\sigma$ 又是复杂度旋钮：$\sigma = 0.03$ 时训练全对、测试 0.671，每个点都是 support vector —— 在背训练集；$\sigma = 10$ 时所有点之间的核值都接近 1，$C = 1$ 下模型几乎弯不动，欠拟合（我们本以为它会”退化成线性 SVM”，实际上 0.671 比线性的 0.5705 好不少，更准确的描述是”欠拟合”）。
• The number of support vectors is a rough complexity meter: close to $n$ usually means over- or underfitting ($C$ or $\sigma$ badly chosen). The classical leave-one-out bound is LOO error $\le \#\text{SV}/n$ (Exercise 6).

5.5 Soft margin, C and the hinge loss

Real data are rarely separable. Allow violations with slacks $\xi_i \ge 0$ and pay $C$ per unit:

The right-hand form is Lecture 1 §3.4’s promise: the SVM is hinge loss + L2 regularization — the same “loss + regularizer” template as ridge or regularized logistic regression, with a different loss. In the dual the only change is a box: $0 \le \alpha_i \le C$. KKT now sorts points into three groups: $\alpha_i = 0$ (outside the margin), $0 < \alpha_i < C$ (exactly on it), $\alpha_i = C$ (inside it or misclassified).

演示 2（两团重叠的高斯，训练 200、测试 5000，特征先中心化）：

• Small $C$ = strong regularization: a very wide margin, with almost every point inside it and a support vector. Large $C$: the margin narrows to 1.70 and then stops changing — the data aren’t separable, the minimum hinge loss is finite, and a larger $C$ only approaches that minimizer.
• 这份二维数据的 Bayes 边界就是一条直线，所有 $C$ 的测试准确率都在 0.860–0.867，正则化的强弱影响不大。$C$ 真正重要的是高维、噪声大的数据（第 3 讲）。
• The relative duality gap (primal − dual)/primal is below $10^{-8}$ for every $C$. Larger $C$ makes SMO slower: many $\alpha_i$ have to creep up to $C$ in small steps. (An earlier version also had $C = 1000$: 780,013 pair updates, with every reported quantity (#SV, margin width, accuracy) identical to $C = 100$, so it was dropped to keep the runtime short.)

5.6 SMO: why two variables at a time

The dual is a QP with $n$ variables; generic QP solvers need $O(n^2)$ memory and are slow. Sequential Minimal Optimization (Platt 1998) is coordinate ascent with a twist. 只改一个 $\alpha_i$ 不行：等式约束 $\sum_i\alpha_i y^{(i)} = 0$ 会把它钉死（其他的 $\alpha$ 不动时 $\alpha_i$ 是确定的）。所以每次改一对 $(\alpha_i, \alpha_j)$：沿着保持 $\alpha_i y^{(i)} + \alpha_j y^{(j)}$ 不变的方向移动，目标函数是一元二次函数，最优步长有封闭解，再截断到 $[0, C]$ 的盒子里。整个子问题不需要任何数值优化器。

怎样选这一对？CS229 讲义的”简化版 SMO”随机选第二个；smo_train 用 LIBSVM 的 maximal violating pair（Keerthi et al. 2001）：选 KKT 条件被违反得最厉害的一对，违反程度小于容差就停止。每步更新梯度只要 $O(n)$。

反过来看：如果没有截距 $b$，对偶里就没有等式约束，可以一次只改一个 $\alpha_i$ —— 这就是 LIBLINEAR 的 dual coordinate descent（Hsieh et al. 2008），代码里的 dcd_svm_nobias。§5.7 会用到它。

5.7 SVM vs logistic regression, and the payoff of Lecture 1 §3.5

演示 2 的最后一行：同样 $\lambda$ 的逻辑回归和 SVM 的测试准确率最多差 0.0022。（初版里 $C = 0.001$ 时逻辑回归明显更差 —— 第 1 讲的实现连截距一起惩罚，SVM 的 $b$ 不受惩罚；不中心化时逻辑回归掉到 0.7612。比较两个模型时，先确认它们解的是同一个问题。）

The payoff. Lecture 1 §3.5 showed that on separable data unregularized logistic regression has no MLE: $\lVert\theta\rVert\to\infty$, but the direction $\theta/\lVert\theta\rVert$ seemed to settle, and we quoted Soudry et al. (2018): GD converges in direction to the max-margin solution. 演示 4 兑现这个承诺，用的是第 1 讲同一份可分数据、同一个 GD（步长 $1/L$，$\lambda = 0$；$10^5$ 步时 $\lVert\theta\rVert = 11.897$，和第 1 讲逐位相同）：

“不加正则化跑很久的 GD”和”$\lambda \to 0$ 的正则化路径”（Rosset, Zhu & Hastie 2003）走到同一个方向 —— 优化算法本身在做正则化，这就是第 3 讲的 implicit regularization。

一个要说清楚的细节：哪一个 SVM？ 第 1 讲把截距当作 $x$ 的一个分量，GD 对 $[\theta_0, \theta_1, \theta_2]$ 一视同仁。Soudry 的定理说的是增广空间里的 max margin：

截距也被惩罚了，和 §5.2 的标准 SVM 不是同一个问题。在第 1 讲那份数据上两者恰好相同（夹角 0.0000°）：3 个 support vectors 给出 3 个等式，正好钉死 3 个参数 $[b, w_1, w_2]$。为了看到差别，演示 4(b) 手工构造了 6 个点 —— 负类 (1,2) (0,4) (0,0)，正类 (3,2) (4,4) (4,0)：

• 标准 SVM 是竖线 $x_1 = 2$，几何间隔 1，只有 2 个 support vectors，$b = -2$；
• 截距被惩罚的 SVM 宁可把间隔降到 0.894，换一个更小的 $\lvert b\rvert$：$[b, w] = [-1, 1, -0.5]$，和标准 SVM 差 26.6°。

GD 明确地走向前者（$\lambda = 10^{-8}$ 的 L2 逻辑回归也是：0.97° vs 25.7°）。所以准确的说法是：GD 最大化的是增广空间 $[1, x]$ 里的间隔，它依赖于坐标原点放在哪里；只有截距不进入 $\lVert\cdot\rVert^2$（不被惩罚），才对应教科书上的 SVM。

收敛有多慢？ Soudry et al. 证明方向误差一般只按 $O(1/\log t)$ 下降。(b) 里从 $10^4$ 到 $10^6$ 步夹角缩小 1.46 倍，$1/\log t$ 预测的是 $\log 10^6/\log 10^4 = 1.5$ 倍 —— 吻合。(a) 在 $10^3$–$10^6$ 步之间每 10 倍步数缩小 2–5 倍，比 $1/\log t$ 快得多：$O(1/\log t)$ 是一般情况下的速度，具体数据的几何可以更友好。两种情况都说明：别指望靠”多跑几步”得到 SVM，要 max margin 就直接解 SVM。

6. 小结 · Summary

Five takeaways:

1. Generative vs discriminative is a bias–variance trade. A correct generative model is more data-efficient (logistic regression made 1.2–1.3× as many excess errors as GDA); a wrong one is biased forever (GDA stuck at 0.136 vs 0.100). Logistic regression assumes strictly less.
2. The sigmoid has three origins: a squashing function, the canonical Bernoulli GLM response, and the exact posterior of shared-covariance Gaussians (or any exponential-family class-conditionals).
3. Naive Bayes classifies well and estimates probabilities badly: double-counted evidence scales the log-odds without flipping its sign. Smoothing is a prior, and $\alpha$ is a hyperparameter.
4. The kernel trick needs two things: an algorithm that only uses inner products, and a solution in the span of the data. Valid kernels are exactly those with PSD Gram matrices; the cost moves from $D$ to $n$.
5. SVM = hinge loss + L2 = maximum margin. KKT gives sparsity (support vectors) and kernelization for free; and GD on logistic regression secretly heads toward a max-margin solution — in the coordinates you gave it, and slowly.

7. 常见误区 · Common pitfalls

• “GDA and logistic regression are the same model.” Same hypothesis class, different estimators (joint vs conditional likelihood): different $\theta$ on the same data, different data efficiency, different robustness.
• “Generative models are always more data-efficient.” Only asymptotically and only when the assumptions hold. With 10 points per class in 10 dimensions, GDA lost to logistic regression (0.229 vs 0.202).
• “Naive Bayes outputs probabilities.” It outputs very confident numbers. With correlated features they are badly overconfident (93.5% of predictions “over 99.9% sure”, only 96% of them right).
• “Laplace smoothing means add one.” $\alpha$ is a prior strength. Here the log loss was best at $\alpha = 0.3$, and $\alpha = 10$ wiped out the signal.
• “Any similarity function is a kernel.” The box kernel and $\tanh$ are intuitive similarities with negative eigenvalues. Check, or build kernels from sums and products of known ones.
• “RBF kernels are smooth, so they can’t overfit.” Small $\sigma$ memorizes the training set (train 1.000, test 0.671 on the circles).
• “The SVM decision value is a probability.” It’s a scaled distance to the boundary; calibrate with Platt scaling if you need probabilities.
• “Run GD long enough and you get the SVM.” You get the max-margin solution in the augmented space (intercept penalized), at a rate that can be $O(1/\log t)$.
• “Kernel methods scale like linear models.” The Gram matrix is $n \times n$; beyond ~$10^5$ points you need approximations.

8. 练习 · Exercises

练习 1（GDA 的边界与参数个数）. (a) 证明两类共享 $\Sigma$ 时 GDA 的决策边界是超平面；两类各有 $\Sigma_0 \ne \Sigma_1$（QDA）时是二次曲面。(b) 在 $d$ 维里，GDA、QDA、逻辑回归各有多少个参数？$d = 10$ 时分别是多少？这和 §2.4 第一列”GDA 在每类 10 个样本时更差”有什么关系？

(a) log-odds $= \log p(x \mid 1) - \log p(x \mid 0) + \text{const}$。高斯的对数密度里 $x$ 的二次项是 $-\frac12 x^\top\Sigma_c^{-1}x$；共享 $\Sigma$ 时两项相减为 0，只剩线性项，边界 $\{\text{log-odds} = 0\}$ 是超平面。$\Sigma_0 \ne \Sigma_1$ 时剩下 $-\frac12 x^\top(\Sigma_1^{-1} - \Sigma_0^{-1})x$，边界是二次曲面。(b) GDA：$1 + 2d + d(d+1)/2$；QDA：$1 + 2d + d(d+1)$；逻辑回归：$d + 1$。$d = 10$ 时分别是 76、131、11。GDA 要估计的参数是逻辑回归的 7 倍，虽然每个参数（均值、协方差）都很容易估计，样本极少时 $\hat\Sigma$ 的误差会经过 $\hat\Sigma^{-1}$ 放大到 $\theta$ 上。QDA 更甚：每类 10 个样本时 $\hat\Sigma_c$ 只有 9 个自由度，在 10 维里是奇异的。

练习 2（Exponential-family class-conditionals）. Suppose $p(x \mid y = c) = h(x)\exp\big(\eta_c^\top T(x) - a(\eta_c)\big)$ for $c = 0, 1$, with the same $h$ and $T$. Show that $p(y = 1 \mid x) = \sigma(\theta^\top T(x) + \theta_0)$ and find $\theta, \theta_0$. Check that GDA (with known shared $\Sigma$) and the Poisson example of §2.3 are special cases.

log-odds $= (\eta_1 - \eta_0)^\top T(x) - a(\eta_1) + a(\eta_0) + \log\frac{\phi}{1-\phi}$，$h(x)$ 抵消。所以 $\theta = \eta_1 - \eta_0$，$\theta_0 = a(\eta_0) - a(\eta_1) + \log\frac{\phi}{1-\phi}$。GDA：$T(x) = x$、$\eta_c = \Sigma^{-1}\mu_c$、$h(x) \propto e^{-x^\top\Sigma^{-1}x/2}$（共享 $\Sigma$ 才能让 $h$ 相同）、$a(\eta_c) = \frac12\mu_c^\top\Sigma^{-1}\mu_c$，和 §2.3 的公式一致。Poisson：$T(x) = x$、$\eta_{c,j} = \log\lambda_{c,j}$、$a = \sum_j\lambda_{c,j}$，得 $\theta_j = \log(\lambda_{1j}/\lambda_{0j})$、$\theta_0 = \log\frac{\phi}{1-\phi} - \sum_j(\lambda_{1j} - \lambda_{0j})$，正是代码 01 演示 1(b) 用的公式。逻辑回归对这一整族生成模型都是”正确”的，这是它更稳健的根本原因。

练习 3（Smoothing as a prior）. (a) Show that with a Dirichlet($\alpha, \dots, \alpha$) prior on the word distribution $P(\cdot \mid c)$, the posterior mean of $P(w \mid c)$ is $\frac{\text{count}(w, c) + \alpha}{N_c + \alpha V}$. (b) In demo 2 each class has about 500 training words and $V = 300$. What fraction of the “total evidence” is pseudo-counts when $\alpha = 1$? When $\alpha = 10$? Relate this to the accuracy column.

(a) Dirichlet 是多项分布的共轭先验：后验是 Dirichlet($\alpha + \text{count}(w_1, c), \dots$)，Dirichlet($\beta$) 第 $w$ 个分量的均值是 $\beta_w/\sum_v\beta_v$，代入即得。（MAP 是 $\frac{\text{count} + \alpha - 1}{N_c + (\alpha - 1)V}$，所以”加 $\alpha$“也可以看成 Dirichlet($\alpha + 1$) 先验下的 MAP。）(b) 伪计数 $\alpha V$：$\alpha = 1$ 时 300，占 $300/800 \approx 38\%$；$\alpha = 10$ 时 3000，占 $3000/3500 \approx 86\%$ —— 估计出的词分布几乎就是均匀分布，信号被抹平，准确率掉到 0.6945。词表越大、训练集越小，同一个 $\alpha$ 的正则化越强，这也是 log loss 在 $\alpha = 0.3$ 而不是 1 处最好的原因。

练习 4（Double-counted evidence）. In demo 3 every word is replaced by $k$ always-co-occurring copies. Show that with multinomial NB and Laplace smoothing $\alpha$, each copy’s estimated probability is exactly $P(w \mid c)/k$ (where $P(w \mid c)$ is the $k = 1$ estimate with the same $\alpha$), and that the log-odds of every document is multiplied by exactly $k$ when the classes are balanced. Why does accuracy stay identical while log loss grows?

复制后每个副本的计数都是 $\text{count}(w, c)$，类的总词数变成 $kN_c$，词表变成 $kV$：$\frac{\text{count}(w,c) + \alpha}{kN_c + \alpha kV} = \frac1k P(w \mid c)$。一封文档里每个副本出现 $x_w$ 次，log 似然 $= \sum_w k\,x_w\big(\log P(w \mid c) - \log k\big)$；$-k\log k\sum_w x_w$ 与类别无关，在 log-odds 里抵消，剩下 $k \times$（原来的 log-likelihood ratio）。先验相等时 log-odds 恰好乘以 $k$。符号不变 → 分类不变；但 $\sigma(k\,z)$ 随 $k$ 越来越接近 0/1，错的那些样本被”自信地”判错：一个分错的样本的损失是 $\log(1 + e^{k|z|}) \approx k|z|$，随 $k$ 大致线性增长；分对的样本损失只能降到 0，抵不过这一项。