Lecture 3 · 泛化、正则化与模型选择 · Generalization, Regularization & Model Selection

对应 Maps to：Stanford CS229 Autumn 2018 Lectures 8、9、13（Data Splits, Models & Cross-Validation; Approx/Estimation Error & ERM; Debugging ML Models and Error Analysis），CS229 main notes（Andrew Ng & Tengyu Ma）Ch. 8（Generalization）和 Ch. 9（Regularization and model selection），CS229 learning theory notes，以及 Andrew Ng 的讲稿 “Advice for applying machine learning”。 CS229 当前学期（Summer 2026）的课表需要 Stanford 登录才能看，所以本讲以公开的讲义和 2018 年的录像为准；这部分内容多年来基本没变（double descent 和 implicit regularization 是新版 main notes 加进来的）。 配套代码 Code：code/lecture03/（numpy；1 个工具模块 + 5 个脚本，最慢的约 16 秒） 预计阅读时间：2.5–3 小时（含运行代码）。这是一份复习讲义：重点放在”为什么”和”实验里到底看到了什么”上，包括好几处和教科书故事不完全一样的结果。

0. 本讲导航 · How to read this lecture

0.1 Learning goals

After this lecture you should be able to:

1. 写出 bias–variance 分解，并说清楚 bias 和 variance 是”对训练集的随机性取期望”才有定义的量 —— 以及为什么高次多项式的 variance 是重尾的。
2. Explain double descent with one picture in your head: the smallest singular value of the feature matrix, and the minimum-norm solution chosen by lstsq / GD-from-zero (implicit regularization).
3. 用 Hoeffding + union bound 推出有限假设类的样本复杂度 $m = O\big(\frac{1}{\gamma^2}\log\frac{k}{\delta}\big)$，说出 VC 维在替换 $\log k$ 时扮演的角色，以及这些界”松在哪里、对在哪里”。
4. 把 ridge、lasso、early stopping、data augmentation 都看成”先验”：知道它们各自的偏差往哪个方向，以及先验不对时会发生什么。
5. 正确使用交叉验证：k 的选择、LOOCV 的封闭公式、one-standard-error rule、选择之后的乐观偏差与 nested CV。
6. 用学习曲线、“optimizer vs objective”诊断、error analysis 和 ablative analysis 决定”下一步该做什么”。

0.2 默认你已经会的 / 本讲会省略的

0.3 本讲地图 Map

§1  Bias–variance：重新采样训练集，直接"看见"两项；approximation vs estimation error       ← 代码 01
§2  Double descent：p = n 的尖峰、最小奇异值、第二次下降、implicit regularization            ← 代码 02
§3  Learning theory：ERM、Hoeffding + union bound、一致收敛、VC 维                           ← 代码 03
§4  Regularization：ridge = MAP、完整后验、lasso、early stopping 与数据增强                   ← 代码 04
§5  Model selection：hold-out / k-fold / LOOCV、选择之后的乐观偏差、nested CV、贝叶斯视角       ← 代码 05 演示 1–2
§6  ML advice：学习曲线、optimizer vs objective、error analysis、ablative analysis           ← 代码 05 演示 3–6
§7–11 小结、常见误区、练习、延伸阅读、下一讲预告

0.4 Notation（在第 1 讲的基础上新增）

代码的约定（和第 1 讲一致）：回归的代价写成平均形式 $J(\theta) = \frac{1}{2n}\lVert X\theta - y\rVert^2 + \frac{\lambda}{2}\lVert\theta\rVert^2$，所以 $\lambda$ 的含义不随样本量变化；截距不惩罚。

0.5 关于数字

讲义里所有实验数字都来自配套代码的实际输出（固定随机种子，同一台机器上重复运行逐字相同）。数据是合成的：我们知道真实函数，所以 bias、variance、泛化误差都能精确地算或无限地重新采样 —— 这正是本讲需要的。好几处结果和”预想的故事”不一样，讲义按实际结果写，并在代码 README 的”关于实验结果”里记录了原计划和改动的原因。

1. Bias–variance · 偏差与方差

1.1 The decomposition

Fix a test point $x$. The learned model $\hat f_S$ depends on the random training set $S$; let $\bar f(x) = \mathbb{E}_S[\hat f_S(x)]$ be the “average model”. For a fresh label $y = f(x) + \epsilon$:

The key step: write $y - \hat f_S = \epsilon + (f - \bar f) + (\bar f - \hat f_S)$ and square. The cross terms vanish because $\epsilon$ is independent of $S$ with mean zero, and $\mathbb{E}_S[\bar f - \hat f_S] = 0$ by definition. So this is an identity, not an approximation.

直觉：bias 是”平均而言错了多少”—— 假设类太小、或者正则化把模型拉偏了；variance 是”换一个训练集，预测会抖多少”—— 模型太灵活，把噪声也学进去了。两者都是对训练集的随机性取期望才有定义的量。现实中你只有一个训练集，看不到它们；合成数据可以无限地重新采样训练集，于是两项都能直接算出来。第 1 讲 §2.5 只用了一个训练集，这里是它的续集。

1.2 Seeing it: polynomial degree

代码 01_bias_variance.py 演示 1：$y = \sin(2\pi x) + \mathcal{N}(0, 0.3^2)$，$n = 30$，每个次数 $k$ 重新采样 $R = 2000$ 个训练集，在 1000 个固定的测试点上算 bias² 和 variance：

What to notice:

• The identity checks out: the formula row and the directly measured row agree to about 0.1%. Bias falls with $k$, variance rises, and the sum is U-shaped with its minimum at $k = 3$.
• $k = 2$ is worse than $k = 1$, and $k = 4$ worse than $k = 3$. In the centred coordinate $u = 2x - 1$, $\sin(2\pi x) = -\sin(\pi u)$ is odd, so even powers cannot reduce the bias (0.199 → 0.202) but still cost an extra parameter’s worth of variance. “One more parameter” is never free.
• The variance is heavy-tailed. For $k \ge 6$ the mean and the median part ways ($k = 9$: mean 30.9, median 0.193). Most training sets are fine; a few leave a gap near $x = 0$ or $x = 1$, the polynomial extrapolates into the gap and explodes, and those rare sets dominate the expectation. 第 1 讲那一个训练集看到的是”典型”的情形；“期望测试误差”却被倒霉的训练集主导。（$k \ge 6$ 的 bias² 不再单调下降，是因为平均预测 $\bar f$ 本身只是 2000 个样本的平均，混进了约 variance$/R$ 的蒙特卡洛误差。）

最初的计划是直接沿用第 1 讲的 $n = 15$。试跑时 $k \ge 5$ 的平均值在两次独立的试跑之间就相差好几倍 —— 期望本身被重尾主导、极不稳定（记录见代码 README 的”关于实验结果”）。所以这里改用 $n = 30$，并同时报告中位数。

1.3 The same trade-off with a different knob: ridge

Now fix the flexible model of Lecture 1 — degree 14, $n = 15$, i.e. 15 parameters for 15 points — and add $\frac{\lambda}{2}\lVert\theta_{1:}\rVert^2$ (演示 2，2000 个训练集)：

• 第 1 讲 §2.5 里 $k = 14$ 的测试 MSE 是 $2\times10^9$ —— 这里的中位数 $2.7\times10^9$ 说明那是一个典型的训练集，不是运气差。
• 一点点 $\lambda$ 就天翻地覆：$\lambda = 10^{-6}$ 已经把平均误差从 $10^{21}$ 拉到 23.6。$\lambda = 0.01$ 时最好（0.212，这也回答了第 1 讲 README 的”动手改一改 3”）。再往上，bias 主导；$\lambda = 10$ 时 bias² = 0.486，几乎就是只剩截距的常数模型（演示 1 中 $k = 0$ 的 0.490）。
• $\lambda$ 和 $k$ 是同一个”复杂度旋钮”的两种形式：ridge 不删特征，而是让 $\lVert\theta\rVert$ 变不大，曲线就没法在点之间剧烈摆动。但最好的 ridge（0.212）仍然远不如 $n = 30$ 时的最好模型（0.112）：正则化能控制 variance，不能凭空变出数据。

1.4 Approximation vs estimation error

Lecture 9 of the 2018 course frames the same trade-off from the learning-theory side. Let $h^\star_k$ be the best degree-$k$ polynomial (the $L^2$ projection of $f$ onto the class) and $\hat h$ the least-squares fit. Then

exactly — a Pythagorean identity, because $\hat h - h^\star_k$ lies in the class and $h^\star_k - f$ is orthogonal to it. 演示 3（每格 400 个训练集；括号里是经验公式 $(k+1)\sigma^2/n$）：

• Approximation error depends only on the class; more data never reduces it. Estimation error falls like $1/n$ and, once $n \gg k$, matches the rule of thumb “each parameter costs $\sigma^2/n$”.
• 模型错设时规则会失效：$k = 1$ 的 estimation error 是公式的 3.5 倍，因为逼近不了的那部分 $f - h^\star$ 对估计来说也像噪声（有效噪声 ≈ $\sigma^2 + 0.196$）。$n = 30$、$k \ge 5$ 又是重尾。
• The optimal complexity grows with $n$ (3 → 5): with more data you can afford to pay the estimation cost of more parameters. 这是”模型应该随数据一起长大”的定量版本 —— 也是今天 scaling laws 的古典祖先。
• 和 bias–variance 的关系：bias² ≈ approximation error，variance ≈ estimation error。前者以”平均模型”为参照，后者以”类里最好的模型”为参照；后者是 §3 学习理论的语言。完整地写：$\varepsilon(\hat h) =$ Bayes error $+$ approximation $+$ estimation。

2. Double descent & implicit regularization · 双下降与隐式正则化

2.1 The puzzle

§1 的 U 形曲线说：过了最佳点，参数越多 variance 越大。可是现代神经网络的参数远多于训练样本，连随机标签都能背下来（Zhang et al., 2017），却照样泛化得很好。Belkin et al.（2019）发现：容量继续增大、越过插值阈值（interpolation threshold）之后，测试误差往往会再次下降 —— 这就是 double descent。新版 CS229 讲义（Ch. 8.2）用线性模型讲它，我们也一样：随机 ReLU 特征 $\phi(x) = \max(0, W^\top x)/\sqrt{p}$，$W$ 随机生成后冻结（相当于只训练最后一层的两层网络），用 lstsq 拟合 —— $p > n$ 时它返回最小范数的插值解（第 1 讲 §2.3）。

2.2 The curve

代码 02_double_descent.py 演示 1：$x \sim \mathcal{N}(0, I_{10})$，$y = u^\top x + \frac12((v^\top x)^2 - 1) + \mathcal{N}(0, 0.3^2)$，$n = 100$ 固定，每个 $p$ 重复 50 次（只预测 0 的模型测试误差是 1.59）：

• First descent and a classical U for $p < n$ (best at $p = 40$), a spike exactly at $p = n$, then a second descent: training error is zero for every $p > n$, yet test error falls monotonically to 0.448 at $p = 3000$ — better than the classical sweet spot.
• At the spike, $s_{\min}$ collapses and $\lVert\theta\rVert$ explodes; past it, both recover. 参数越多，可供选择的插值解越多，最小的那个范数越小。
• A fixed ridge removes the spike entirely — but this particular $\lambda$ is too large for big $p$ (0.702 vs 0.448). $\lambda$ has to be tuned (§2.4).

2.3 Why $p \approx n$ is the worst place to be

The minimum-norm solution is $\theta = \Phi^+ y = \sum_i \frac{u_i^\top y}{s_i}\,v_i$: the component of the noise along the $i$-th singular direction is amplified by $1/s_i$. 最干净的版本（演示 2）：标签是纯噪声 $y = \epsilon \sim \mathcal{N}(0, 1)$，特征 iid $\mathcal{N}(0,1)$，于是测试误差 $-\,\sigma^2 = \lVert\theta\rVert^2$ 全部是 variance，而且有精确公式：

（$p = 98$、102 时 $1/s^2$ 的分布尾巴很重，400 次的平均也不稳定。）三个区间的直觉：

• $p < n$：每多一个参数，就要用同样多的数据多估一个数 —— 经典的 variance 增长。
• $p = n$：方程组恰好被确定，没有任何余地：为了穿过每一个噪声点，θ 必须沿着几乎奇异的方向走很远。随机矩阵理论说 $s_{\min} \approx \lvert\sqrt n - \sqrt p\rvert$（Bai–Yin），方阵时贴着 0。
• $p > n$：插值解构成一个 $(p - n)$ 维的仿射空间，最小范数解在里面挑最”温和”的；多出来的维度把噪声摊薄了（$n/(p - n - 1) \to 0$）。代价是：真实函数不为 0 时，“偏好小范数”本身会引入 bias —— 所以第二次下降能降到多低，取决于问题。

2.4 Does the second descent beat the first?

演示 3 换三个真实函数（每个 $p$ 重复 50 次；ridge 的 $\lambda$ 按测试集挑 —— oracle，只为展示潜力）：

Three targets, three answers: the overparameterized interpolant beats the classical optimum clearly (lin+quad), slightly (linear), or not at all (tanh). “Interpolation is better” is not a theorem. What is robust: the spike is caused by unregularized exact interpolation, and a tuned ridge removes it in every case (Nakkiran et al., 2021, prove that optimal regularization makes the curve monotone in some settings); the best results all come from “many features + the right $\lambda$”. 实践里很少看到这么夸张的尖峰，正是因为现代训练几乎总带着某种显式或隐式的正则化。真正的教训是：参数个数不是复杂度的好度量，范数（以及优化器偏好哪种解）才是。

设计说明：最初打算在一维的 make_sine 上做。一维 ReLU 特征在训练点上高度共线（拐点落在同一对相邻训练点之间的特征只张成二维空间），试跑时 $p$ 远大于 $n$ 也看不到第二次下降，于是改用 10 维输入。三个 target 都列出来，而不是只挑一个故事最好看的（试跑记录见代码 README）。

2.5 Implicit regularization: the optimizer picks the solution

With $p > n$ there are infinitely many interpolants. Which one GD finds depends on where it starts. Each GD step on least squares adds $-\alpha\,\Phi^\top(\cdot)$, which lies in the row space of $\Phi$, so the null-space component of $\theta$ never changes:

From $\theta_0 = 0$ this is the minimum-norm solution (第 1 讲 §2.3 用 GD 验证过)。演示 4（$n = 100$，$p = 400$，同一份数据和特征）：

All four limits interpolate the training set — the training loss cannot tell them apart — yet test error grows with the distance of the start from zero. We never wrote “minimize $\lVert\theta\rVert$”; starting GD at zero did it for us. The same story holds for classification (第 1 讲 §3.5：GD on separable data converges in direction to the max-margin solution), and for deep nets the initialization scale, optimizer, step size and batch size all bias which solution you get.

两个细节：(1) 这里 GD 没有跑到收敛 —— ReLU 特征全为正，$s_{\max}/s_{\min} \approx 120$，GD 要几十万步。所以演示检验的是一个每一步都精确成立的事实（零空间漂移 $\le 4\times10^{-13}$），极限用公式计算。(2) 跑了 2 万步、还没收敛的 GD，测试误差（0.593）比它的极限（0.598）还低一点：GD 先拟合大奇异值方向，最后才去拟合 $s_{\min}$ 方向上的噪声 —— 这就是 §4.4 的 early stopping。

3. Learning theory · 学习理论（直觉为主）

3.1 ERM and the question

Classification with 0-1 loss: $\varepsilon(h) = P_{(x,y)\sim\mathcal{D}}(h(x) \ne y)$ and $\hat\varepsilon(h) = \frac1m\sum_i 1\{h(x^{(i)}) \ne y^{(i)}\}$. Empirical risk minimization picks $\hat h = \arg\min_{h\in\mathcal{H}} \hat\varepsilon(h)$. We want $\varepsilon(\hat h)$ close to $\varepsilon(h^\star)$.

微妙之处在于：对一个固定的 $h$，$\hat\varepsilon(h)$ 是 $m$ 个独立 Bernoulli 变量的平均，会集中在 $\varepsilon(h)$ 附近。但 $\hat h$ 是看过数据之后才挑出来的，$\hat\varepsilon(\hat h)$ 不再是独立变量的平均 —— 它系统性地偏低（正是 §5.2 的选择偏差）。解决办法是要求一致收敛（uniform convergence）：对 $\mathcal{H}$ 里所有的 $h$ 同时有 $\hat\varepsilon(h) \approx \varepsilon(h)$。这样无论 ERM 选中谁，它的训练误差都可信。

3.2 Hoeffding: one hypothesis

代码 03_learning_theory.py 演示 1 把它和精确的二项分布尾概率比较（$\gamma = 0.1$）：

The bound always holds and has the right exponential rate, but it is loose (useless for $m \le 25$; 23× too big at $m = 1000$) because it only uses “each variable lies in [0, 1]” — it must also cover the worst case $\phi = 1/2$ and ignores the variance. 推论：$m \ge \log(2/\delta)/(2\gamma^2)$；$\gamma = 0.01$、$\delta = 0.05$ 需要 18445 个样本 —— 这也是”测试集要多大”的经验法则：精度提高 10 倍，样本多 100 倍。

3.3 Finite $\mathcal{H}$: union bound ⇒ sample complexity

Apply Hoeffding to each of the $k$ hypotheses and add the failure probabilities (union bound): $P(\exists h: \lvert\hat\varepsilon(h) - \varepsilon(h)\rvert > \gamma) \le 2k\,e^{-2\gamma^2 m}$. Setting this to $\delta$:

The ERM guarantee is three lines: $\varepsilon(\hat h) \le \hat\varepsilon(\hat h) + \gamma \le \hat\varepsilon(h^\star) + \gamma \le \varepsilon(h^\star) + 2\gamma$. Note the shape: a bigger $\mathcal{H}$ lowers $\varepsilon(h^\star)$ (approximation) and raises $\gamma$ (estimation) — §1.4 again, now as a theorem. And $k$ enters only through $\log k$: a class 1000 times bigger only adds $\log 1000 \approx 6.9$ to the numerator.

演示 2 用一个真实误差能精确计算的问题检验：$x \sim U(0,1)$，$y = 1\{x > 0.3\}$ 再以 10% 概率翻转；$\mathcal{H}_k$ = $k$ 个等间距阈值 $h_t(x) = 1\{x > t\}$，$\varepsilon(t) = 0.1 + 0.8\lvert t - 0.3\rvert$。$m = 100$，每个设置 2000 次：

上界处处成立，但它随 $\log k$ 一直增长，实际偏差却在 $k \approx 100$ 之后停在 0.135 左右。union bound 把 $k$ 个坏事件当成互不相关的来相加；而相邻的阈值几乎做同样的预测，偏差高度相关，“有效”的假设个数远小于 $k$。固定 $k = 1001$、把 $m$ 从 25 增加到 1600，$m$ 每乘以 4，$\gamma$ 和实际偏差都减半（$1/\sqrt m$ 的速率是对的），两者之比稳定在 1.6–1.7。学习理论的界应该这样读：看它的 scaling，而不是它的常数。

3.4 VC dimension: counting what an infinite class can actually do

A set of points is shattered by $\mathcal{H}$ if every one of the $2^n$ labelings is realized by some $h \in \mathcal{H}$; the VC dimension is the size of the largest shatterable set. 演示 3 精确枚举平面上半平面（2-D 线性分类器 + 截距）能实现的标注：

• VC dim of half-planes is 3: some 3 points are shattered (collinear ones are not — the definition only needs existence), and no 4 points are (Radon’s theorem). In general, linear classifiers in $\mathbb{R}^d$ have VC dimension $d + 1$ = number of parameters. 阈值类 $1\{x > t\}$ 的 VC 维是 1，这解释了 §3.3 的饱和。
• Sauer’s lemma: if VC$(\mathcal{H}) = d$, then on $n$ points $\mathcal{H}$ realizes at most $\sum_{i\le d}\binom{n}{i} = O(n^d)$ labelings — polynomial, not $2^n$. The counts match Cover’s (1965) formula exactly.
• 于是 union bound 里的 $\log k$ 可以换成 $\log O(m^d) \approx d\log m$，得到 Vapnik 的结果：以 $1 - \delta$ 的概率

样本复杂度和 VC 维成正比。对很多模型 VC 维 ≈ 参数个数，所以 Ng 课上的经验法则是”需要的样本数大致和参数个数成正比”（练习 3 给出另一个推导：把参数当成 64 位浮点数，$\lvert\mathcal{H}\rvert = 2^{64p}$）。

3.5 What the classical theory does and does not explain

它解释了：$m$ 远大于模型容量时，训练误差为什么可以当作测试误差的代理；为什么 $\log\lvert\mathcal{H}\rvert$ 或 VC 维是衡量复杂度的正确”货币”；为什么 ERM 是一致的（consistent）。它解释不了 §2：一个 $10^9$ 参数的网络在 $10^6$ 个样本上训练，VC 界是空洞的（vacuous），它却照样泛化。现代理论把”参数个数”换成依赖于实际找到的那个解的量 —— 范数和间隔（§2 的最小范数解、第 1 讲的 max-margin 方向）、PAC-Bayes 界，以及研究”插值带噪声的数据什么时候无害”的 benign overfitting（Bartlett et al., 2020）。一句话：经典理论把复杂度算在假设类上，现代理论把它算在算法实际找到的解上。

4. Regularization · 正则化

4.1 Ridge = MAP with a Gaussian prior — and what that prior assumes

With $y \mid x \sim \mathcal{N}(\theta^\top x, \sigma^2)$ and prior $\theta_j \sim \mathcal{N}(0, \tau^2)$, the negative log-posterior is $\frac{1}{2\sigma^2}\lVert X\theta - y\rVert^2 + \frac{1}{2\tau^2}\lVert\theta\rVert^2$; multiplying by $\sigma^2/n$ gives our ridge objective with $\lambda = \sigma^2/(n\tau^2)$. Regularization = a prior; strong regularization = a confident prior. In SVD coordinates ridge multiplies the least-squares coefficient along the $i$-th direction by $s_i^2/(s_i^2 + n\lambda)$: it shrinks the low-variance directions of the data most, not each coefficient toward zero.

第 1 讲 §2.4 留了一个问题：面积和卧室数相关 0.88，卧室系数的估计方差很大（标准差约 4，真值才 10）。代码 04_regularization.py 演示 1 用和第 1 讲完全相同的 2000 份数据加 ridge（特征标准化，$\lambda = 0$ 一行精确复现第 1 讲）：

原计划的故事是”ridge 治好共线性”；实际结果是：在第 1 讲的真值下，ridge 让卧室系数更差（RMSE 3.95 → 12.49）。方差确实下降了（3.95 → 2.56），但卧室系数的平均值被往上拉到 22。原因就在上面那句话：共线时方差最小的方向是”两个标准化系数之差”，ridge 把它们往一起拉。标准化后的真系数是 [111, 14.6]，相差悬殊 —— 先验”各系数差不多大、都在 0 附近”在这里是错的，偏差的方向恰好不对。对照实验把卧室的真系数换成 80（标准化后 [111, 117]，几乎相等），先验对了，ridge 才帮上忙（3.95 → 3.66）；即便如此改善也不大：$n = 200$、只有 2 个特征，方差本来就不是主要矛盾。第 1 讲那份不走运的样本被 ridge 从 −3.32 拉到 14.92（$\lambda = 0.1$），看起来”修好了”，其实只是收缩方向碰巧和这份样本的误差方向相反。

Lesson: ridge is not a free fix for collinearity. It helps when variance dominates (many features, few samples — §1.3, §2.4, §4.3) and the prior is roughly right. Its bias is along the data’s weak directions, which is exactly where the data can’t tell you the prior is wrong.

4.2 MAP vs the full posterior

The posterior here is Gaussian, so its mode (MAP = ridge) equals its mean — but the posterior also has a covariance $\Sigma_{\text{post}} = \sigma^2(\tilde X^\top\tilde X + \frac{\sigma^2}{\tau^2}D)^{-1}$, which gives an uncertainty for every prediction. A full Bayesian predicts with $p(y \mid x, S) = \int p(y \mid x, \theta)\,p(\theta \mid S)\,d\theta$ instead of plugging in one $\theta$. 演示 2（一份新的房价数据，相关系数 0.89；表中是预测均值及其标准差，单位千美元）：

• Same model, very different confidence: the “big house with one bedroom” is 6× less certain than a typical house — the training data barely constrain the coefficients in that direction. A MAP point estimate throws this information away.
• The extrapolated house (6000 ft², far outside the data) is more certain than the 3800 ft² one-bedroom house, because it lies along the data’s main direction. Uncertainty is about how far you move in directions the data don’t constrain, not about distance from the data.
• 先验错了，后验会自信地错：$\tau = 10$ 时”大而卧室少”被拉到 452.6，离真值偏了 4.2 个后验标准差。不确定性估计只在模型假设（包括先验）合理时才可信。
• 为什么还要贝叶斯？主动学习、Bayesian optimization、“模型什么时候该说’我不知道‘“都需要它。线性高斯有封闭解；神经网络只能近似（Laplace、变分推断、deep ensembles —— 第 4 讲的 bagging 是最朴素的版本）。

4.3 Lasso and sparsity

Replace the Gaussian prior by a Laplace prior and you get the L1 penalty $\lambda\lVert\theta\rVert_1$. Why does it produce exact zeros? In one dimension, $\min_\theta \frac12(\theta - z)^2 + \lambda\lvert\theta\rvert$ is solved by soft-thresholding $S(z, \lambda) = \text{sign}(z)\max(\lvert z\rvert - \lambda, 0)$: the corner of $\lvert\theta\rvert$ at 0 has subgradient $[-1, 1]$, so as long as the data’s “pull” $\lvert z\rvert$ is below $\lambda$, zero is optimal. Ridge’s $\frac{\lambda}{2}\theta^2$ has no corner; its solution $z/(1 + \lambda)$ shrinks but never hits zero. (Geometrically: the L1 ball is a diamond whose corners lie on the axes.) Coordinate descent applies this 1-D solution to one coordinate at a time — that is glmnet.

演示 3：$n = 50$，30 个相邻相关（$\rho = 0.5$）的特征，只有 5 个非零（[3, −2, 1.5, 1, −1]），$\sigma = 1$（测试 MSE 下限 1）。lasso 路径从 $\lambda_{\max}$（解恰好全为 0）往下扫：

对照（$\lambda$ 都按测试集挑，只比潜力）：OLS（30 个特征）3.68；最好的 ridge 2.065（ridge 没有一个系数恰好为 0）；知道真实支撑集的 oracle OLS 1.08。

• 真相稀疏时 lasso 远好于 ridge（1.19 vs 2.065）：Laplace 先验”大部分系数是 0”说对了。真相是”很多小系数”时结论反过来（练习 4）。
• The prediction-optimal lasso is not the selection-optimal lasso: at $\lambda = 0.080$ it keeps 9 false features. One $\lambda$ has to do two jobs — select and shrink — and a $\lambda$ large enough to drop the false features also shrinks the true coefficients by $\lambda$ (4 of the 5 true coefficients come out smaller than the truth). Relaxed lasso decouples the jobs: select with lasso at a larger $\lambda$ (6 features, 5 true), refit by OLS → 1.08, matching the oracle. But refitting at the small $\lambda$ is worse (1.48): without shrinkage, the false features’ variance comes in at full price.
• 相关特征里 lasso 倾向于挑一个代表，挑哪个对数据很敏感；elastic net（L1 + L2）是常见的折中。

4.4 Regularizers in disguise: early stopping and data augmentation

Early stopping. GD from $\theta = 0$ on least squares, after $t$ steps, has fitted a fraction $1 - (1 - \alpha s_i^2/n)^t$ of the $i$-th singular direction; ridge fits $s_i^2/(s_i^2 + n\lambda)$. Both are filters that pass large-$s_i$ directions and block small ones, with $t \leftrightarrow 1/\lambda$ (roughly $\lambda \approx 1/(\alpha t)$, 练习 2)。演示 4(a)：$k = 14$、$n = 15$ 的多项式，200 个训练集同时跑 GD：

Same U, traversed in the opposite direction; the correspondence is rough (the two filters have different shapes, and the valleys are offset by about one column). This is why “stop when the validation loss stops improving” is the cheapest and most widely used regularizer in deep learning — it needs no change to the loss, just a validation set.

Data augmentation. Add noise $\epsilon \sim \mathcal{N}(0, s^2 I)$ to the (standardized) inputs and average over it:

so OLS on many noisy copies converges to ridge with $\lambda = s^2$. 演示 4(b) 在第 1 讲那份房价数据上：每个样本复制 $K$ 份、加 $\mathcal{N}(0, 0.3^2)$ 噪声，重复 20 次 —— 卧室系数（标准化坐标）的平均值 18.47 / 19.26 / 19.58 / 19.60（$K$ = 1 / 10 / 100 / 1000）趋向 ridge 的 19.62（OLS 是 −4.68），离 ridge 的 RMS 距离按 $1/\sqrt K$ 缩小（6.76 → 0.22）。“对输入的小扰动不敏感”和”惩罚 $\lVert\theta\rVert$“是一回事；dropout（Wager et al., 2013 证明它近似一种自适应的 L2）、随机裁剪、mixup 都是这个思路的推广。反过来也成立：代码里的 fit_ridge 就是往训练集里加 $p$ 行假数据（第 $j$ 行只有第 $j$ 个特征为 $\sqrt{n\lambda}$，标签为 0）来实现 ridge 的。正则化 = 把先验知识写成数据或写成损失，两种写法等价。

5. Model selection · 模型选择

5.1 Hold-out, k-fold, LOOCV

• Hold-out：留出 20–30% 当验证集，在剩下的数据上训练每个候选模型，挑验证误差最小的，再用全部数据重新训练。简单，但浪费数据，而且验证误差本身的方差大。
• k-fold CV：分成 $k$ 折，每折轮流当验证集，平均 $k$ 个验证误差。估计的是”用 $(k-1)n/k$ 个样本训练”的误差：$k$ 小 → 训练集小 → 偏悲观；$k$ 大 → 偏差小，但各折训练集几乎相同，估计的方差可能更大，计算也更贵。
• LOOCV（$k = n$）：一般要训练 $n$ 次；但对 ridge 这样的线性平滑器 $\hat y = Hy$，留一残差恰好是 $(y_i - \hat y_i)/(1 - H_{ii})$，训练一次就够（gen_tools.ridge_loocv；$H_{ii}$ 叫 leverage）。

代码 05_selection_diagnostics.py 演示 1：$k = 14$ 的多项式、$n = 30$，用 CV 在 17 个 $\lambda$（$10^{-7}$ 到 10）里选一个，200 个数据集：

• 典型情况下各种方法都选得不错（中位数 0.135–0.138）。平均值被少数”翻车”主导：CV 选了极小的 λ，测试误差爆炸。CV 为什么看不见？它只在训练输入所在的位置检查预测，而爆炸发生在 $x = 0, 1$ 附近没有训练点的空白处。在演示 (a) 的那个数据集上，LOOCV 在 $\lambda = 3\times10^{-7}$ 处报告 0.097，真实测试误差却是 23.9。
• 2 折偏悲观、偏向更大的 λ —— 在这里反而成了保护（平均 0.179）。
• The one-standard-error rule (ESL §7.10): among $\lambda$‘s whose CV error is within one SE of the minimum, take the simplest. It gives up a little in the typical case (0.146) and has the best mean (0.162). It acknowledges that the CV curve itself is noisy.

5.2 The optimism of the winner, and nested CV

Selecting the best of $M$ noisy scores is itself a fit. 演示 2 用最极端的版本：纯噪声，$n = 50$，$y \sim \mathcal{N}(0,1)$，$M$ 个与 $y$ 无关的特征；候选模型 = 只有截距，或截距 + 一个特征。用 5 折 CV 挑最好的，重新拟合后它的真实 MSE 可以精确算出（$1 + \theta_0^2 + \theta_1^2$）。200 个数据集的平均：

$M = 1000$ 时，被选中的模型看起来”解释了 15% 的方差”，实际上比只预测均值还差。$M$ 个带噪声的估计里取最小值，大约会比典型值低 $s\sqrt{2\log M}$（练习 5）—— 和 §3.3 union bound 里的 $\log M$ 是同一个东西：验证集本身就是一个”有限假设类”的学习问题，你看过的每一个假设都要付费。Nested CV 把”选择 + 重新拟合”当成一个整体算法，在外层循环里评估它，给出诚实的估计（略偏悲观，因为外层每次只用 4/5 的数据）。Kaggle leaderboard 过拟合、在测试集上调参、p-hacking，说的都是这张表。

实践规则：测试集只在最后碰一次。任何包含”选择”（选 λ、选特征、选架构）的流程，都应该报告 nested CV（或一个全新的 hold-out 集）上的数字。

5.3 The Bayesian view of model selection（简述）

Bayesians select models by the marginal likelihood $p(S \mid \mathcal{M}) = \int p(S \mid \theta, \mathcal{M})\,p(\theta \mid \mathcal{M})\,d\theta$: a flexible model spreads its prior over many datasets, so it assigns less probability to the one you observed unless it needs the flexibility — an automatic Occam’s razor. Its large-$n$ approximation is BIC $= -2\log\hat L + p\log n$; AIC $= -2\log\hat L + 2p$ estimates out-of-sample log-likelihood instead. Both penalize parameter count, so both inherit §3.5’s caveat for overparameterized models. In everyday practice CV wins because it makes no assumption about the model being correct.

6. Advice for applying ML · 怎样决定下一步

Ng 在 L13 和 “Advice for applying machine learning” 里的核心论点：与其凭直觉试（更多数据？更多特征？换模型？），不如先做诊断，弄清楚问题出在哪。 每一种”修复”只对一种问题有效，用错了就白干几个月。

6.1 Learning curves: high bias or high variance?

Plot training and validation error against training-set size $n$. 演示 3（200 个训练集的中位数；目标是测试 MSE ≤ 0.12，噪声下限 0.09）：

Two reading tips: the gap diagnoses variance, and the training error relative to the target diagnoses bias — if even the training error misses the target, more data cannot help, because training error usually only rises with $n$. Ridge at $\lambda = 10^{-3}$ is “a prior worth some data”: it matters enormously at $n = 20$ (0.139 vs 56.2), and at $n = 640$ it is even slightly worse (0.0946 vs 0.0918) — its small bias now outweighs the variance it removes.

6.2 Is it the optimizer or the objective?

Ng 的例子：用 Bayesian logistic regression（= 带高斯先验的逻辑回归）做垃圾邮件过滤，误删正常邮件的比例高得不可接受；一个线性 SVM 在你真正关心的指标上更好，但出于别的原因你想继续用逻辑回归。问题出在优化算法（没把 $J$ 优化好）还是目标函数（$J$ 本身不对）？诊断只需要一次比较：用 A 自己的目标 $J$ 去评价 B 的参数。

• $J(\theta_B) < J(\theta_A)$（我们的 $J$ 是损失，越小越好）：B 在 A 自己的目标上都赢了 A ⇒ A 的优化器没完成任务。
• $J(\theta_B) \ge J(\theta_A)$：$J$ 认为 A 更好，而真正的指标 $a$ 认为 B 更好 ⇒ $J$ 和 $a$ 不一致 ⇒ 目标函数有问题。

演示 4 的玩具版：误删正常邮件代价 10、漏掉垃圾邮件代价 1，$a$ = 测试集平均代价。A = 不加权的逻辑回归，用 GD 训练（特征没标准化，条件数很差）；B = 正常邮件权重 10 的加权逻辑回归：

Once the optimizer is fixed (Newton), the diagnosis is unambiguous: A minimizes its $J$, yet its cost is twice B’s — unweighted log loss treats both errors as equally bad, and we don’t. Fix the objective (weighted loss, or move the decision threshold), not the optimizer.

A subtlety the toy exposed: the two outcomes are not symmetric. After 50 steps GD is far from converged ($J = 0.3547$ vs optimum 0.1991), yet the diagnosis already says “objective”. "$J(\theta_B) < J(\theta_A)$" proves an optimizer problem; "$J(\theta_B) \ge J(\theta_A)$" proves that $J$ and $a$ disagree, but not that the optimizer is fine — both problems can coexist. 修好一个之后要再诊断一次。（原计划是”50 步 = optimizer 问题”的演示；实际输出不支持，于是改成扫步数，把这个不对称性写了进来。）同样的思路第 6 讲还会用到：RL 里是模拟器、算法还是奖励函数的问题？

6.3 Error analysis and ablative analysis

Error analysis asks: which component is worth improving? For a pipeline, replace one component’s output by ground truth and see how much the final metric improves. 演示 5：组件 A 估计潜变量 $s_1$（噪声 1.0），组件 B 估计 $s_2$（噪声 0.3），最后的逻辑回归预测 $y \sim \text{Bernoulli}(\sigma(3s_1 + 1.5s_2))$：

改进 A 最多能涨 10.8 个百分点，改进 B 最多 0.65 个 —— 即使 B 也不完美，也该先改 A。“哪个组件误差最大”不是正确的问题：组件误差对最终指标的影响取决于下游怎么用它；error analysis 直接量这个影响。Ng 的人脸识别例子（预处理 → 人脸检测 → 眼 / 鼻 / 嘴分割 → 逻辑回归）就是沿着流水线逐个插入真值。

Ablative analysis asks the reverse: which existing components actually help? Remove them and measure the drop. 演示 6：A 和 A′ 是同一个 $s_1$ 的两个独立测量（冗余），B 测 $s_2$，junk 是 10 个噪声特征；全部特征 0.795：

Leave-one-out ablation says B is the most important group — yet $s_1$‘s true coefficient is twice $s_2$‘s. A and A′ cover for each other, so removing either alone barely hurts; removing both costs 17.5 points. Cumulative ablation (Ng’s style: remove one group after another) is order-dependent for the same reason: in the order junk → A′ → B → A, A′ “costs” 2.8 points and A “costs” 22.4. Redundancy makes ablation tables lie: state the order, and report pairwise removals for components that might be redundant.

Ng 的总建议：先快速做一个粗糙的系统（quick-and-dirty），然后用上面的诊断决定把时间花在哪里，而不是一开始就精心设计每个组件（“premature statistical optimization”）。

7. 小结 · Summary

Six takeaways:

1. Bias and variance only exist across training sets. Resampling makes them visible; one training set can’t. And variance can be heavy-tailed — “expected” and “typical” test error can differ by orders of magnitude.
2. Parameter count is a poor measure of complexity. What matters is the norm of the solution and which solution the optimizer prefers (min-norm, max-margin). Exact interpolation at $p \approx n$ is the danger zone, not $p \gg n$.
3. Learning-theory bounds are for scaling, not numbers: $1/\sqrt m$, $\log\lvert\mathcal{H}\rvert$, VC dimension. They are valid and loose.
4. Every regularizer is a prior, and a prior can be wrong. Ridge shrinks along the data’s weak directions; when the truth isn’t small there, it adds bias exactly where the data can’t correct it.
5. Selection is fitting. The best of $M$ validation scores is optimistic by $\sim\sqrt{\log M}$; use nested CV or an untouched test set.
6. Diagnose before fixing. Learning curves, the optimizer-vs-objective test, error analysis and ablation each answer a different “what next?” — and each has failure modes (asymmetry, redundancy).

8. 常见误区 · Common pitfalls

• “Low training error means a good model.” Training error falls monotonically with complexity (§1.2, §6.1). It measures fit, not learning.
• “More parameters always means more overfitting.” Past the interpolation threshold, min-norm solutions can get better with more parameters (§2.2) — though not always better than the classical optimum (§2.4).
• “Ridge fixes collinearity.” It reduces variance but biases correlated coefficients toward each other; with a very unequal truth it made the estimate worse (§4.1).
• “The CV score of the selected model is an estimate of its test error.” It is optimistic; the more models you compared, the more so (§5.2).
• “LOOCV is always the best CV.” It has the least bias but can be unstable, and like all CV it only checks predictions where the training inputs are (§5.1).
• “If the diagnostic says ‘objective’, the optimizer is fine.” The test is asymmetric; both can be broken (§6.2).
• “Removing a component barely hurt, so it’s useless.” Not if another component is redundant with it (§6.3).

9. 练习 · Exercises

练习 1（Bias–variance identity）. (a) Prove the decomposition in §1.1, showing each cross term vanishes. (b) For $k$-nearest-neighbour regression with fixed training inputs, $\hat f(x) = \frac1k\sum_{i\in N_k(x)} y^{(i)}$, show that the variance at $x$ is $\sigma^2/k$ and write the bias. Which way does $k$ move the trade-off?

(a) $y - \hat f_S = \epsilon + (f - \bar f) + (\bar f - \hat f_S)$。三个交叉项：$\mathbb{E}[\epsilon(f - \bar f)] = 0$（$\epsilon$ 均值为 0，$f - \bar f$ 是常数）；$\mathbb{E}[\epsilon(\bar f - \hat f_S)] = \mathbb{E}[\epsilon]\,\mathbb{E}[\bar f - \hat f_S] = 0$（测试噪声和训练集独立）；$\mathbb{E}_S[(f - \bar f)(\bar f - \hat f_S)] = (f - \bar f)\,\mathbb{E}_S[\bar f - \hat f_S] = 0$。(b) $\text{Var}[\hat f(x)] = \frac{1}{k^2}\cdot k\sigma^2 = \sigma^2/k$；bias $= \frac1k\sum_{i\in N_k(x)} f(x^{(i)}) - f(x)$，邻居越远越大。$k$ 大 → variance 小、bias 大：$k$ 和 LWR 的带宽 $\tau$（第 1 讲 §2.5）是同一种旋钮。

练习 2（Early stopping as a spectral filter）. For $J(\theta) = \frac{1}{2n}\lVert\Phi\theta - y\rVert^2$, GD from 0 with step $\alpha$. Show $\theta_t = \sum_i \big[1 - (1 - \alpha s_i^2/n)^t\big]\frac{u_i^\top y}{s_i}v_i$, compare with ridge’s filter $\frac{s_i^2}{s_i^2 + n\lambda}$, and argue why $\lambda \approx 1/(\alpha t)$.

在 $V$ 的坐标下各分量独立：$c_{t+1} = c_t - \frac{\alpha}{n}(s_i^2 c_t - s_i u_i^\top y)$，不动点 $c^\star = u_i^\top y / s_i$，误差每步乘以 $(1 - \alpha s_i^2/n)$，从 $c_0 = 0$ 出发得到上式。令 $a = \alpha s_i^2/n$：GD 的滤波 $1 - (1 - a)^t \approx 1 - e^{-at}$，ridge 的滤波 $\frac{s_i^2}{s_i^2 + n\lambda} = \frac{a}{a + \alpha\lambda}$。两者都在 $a \gg 1/t$（或 $a \gg \alpha\lambda$）时 ≈ 1，在 $a \ll 1/t$ 时 ≈ $at$（或 $a/(\alpha\lambda)$）。令两个转折点重合：$1/t = \alpha\lambda$，即 $\lambda \approx 1/(\alpha t)$。形状不同（指数 vs 有理函数），所以 §4.4 的两条曲线谷底错开了一点。零空间方向（$s_i = 0$）两者都不碰 —— 这就是 §2.5。

练习 3（Parameters as 64-bit floats）. A model has $p$ real parameters, each stored as a 64-bit float. (a) Bound $\lvert\mathcal{H}\rvert$ and give the sample complexity for $\gamma$, $\delta$. (b) Evaluate for $p = 100$, $\gamma = 0.05$, $\delta = 0.05$. (c) Why is this argument both reassuring and unsatisfying?

(a) $\lvert\mathcal{H}\rvert \le 2^{64p}$，所以 $m \ge \frac{1}{2\gamma^2}\big(64p\log 2 + \log\frac{2}{\delta}\big) = O(p/\gamma^2)$。(b) $\frac{1}{2\cdot 0.0025}(6400\times 0.693 + \log 40) = 200\times(4436 + 3.7) \approx 8.9\times10^{5}$。(c) 让人放心：样本复杂度线性于参数个数（和 VC 维的结论一致），这是 Ng 的经验法则的来源。不满意：它依赖于浮点表示这种与学习无关的细节；常数很大；而且对 $p \gg m$ 的现代模型完全失效（§3.5）—— 真正决定泛化的是算法找到的解，不是可表示的假设有多少。

练习 4（When does ridge beat lasso?）. With an orthonormal design ($X^\top X = nI$), the least-squares coefficients are $z_j$; show ridge gives $z_j/(1 + \lambda)$ and lasso gives $S(z_j, \lambda)$. Use this to explain why lasso wins when the truth is sparse with a few large coefficients, and ridge wins when the truth has many small coefficients of similar size. (Code version: 动手改一改 4 in the README.)

正交设计下目标函数按坐标分解：$\frac12(\theta_j - z_j)^2 + \frac{\lambda}{2}\theta_j^2$ 的解是 $z_j/(1 + \lambda)$，$\frac12(\theta_j - z_j)^2 + \lambda\lvert\theta_j\rvert$ 的解是软阈值。稀疏真值：大部分 $z_j$ 只是噪声，软阈值把它们整个去掉（variance 归零），少数大系数只付出固定的 $\lambda$ 偏差；ridge 对每个噪声坐标只按比例缩小，留下 $\propto 1/(1+\lambda)^2$ 的 variance。稠密的小系数：真系数和噪声大小相当，软阈值会把很多真系数也砍成 0（bias = 整个系数），而 ridge 按比例保留每一个的一部分。贝叶斯说法：Laplace 先验押注”大部分为 0”，高斯先验押注”都小但都不为 0”，谁的先验对谁赢。

练习 5（Why $\sqrt{2\log M}$）. Let $Z_1, \dots, Z_M$ be $\mathcal{N}(0, 1)$, not necessarily independent. Show $\mathbb{E}[\max_i Z_i] \le \sqrt{2\log M}$. Relate this to the optimism in §5.2 and to the $\log k$ in §3.3.

对任意 $t > 0$，由 Jensen：$e^{t\,\mathbb{E}\max_i Z_i} \le \mathbb{E}\,e^{t\max_i Z_i} \le \sum_i \mathbb{E}\,e^{tZ_i} = Me^{t^2/2}$，所以 $\mathbb{E}\max_i Z_i \le \frac{\log M}{t} + \frac{t}{2}$，取 $t = \sqrt{2\log M}$ 得证。“把 max 换成 sum”正是 union bound 的思路，所以同一个 $\log M$ 出现在两处。§5.2 里每个候选模型的 CV 分数 = 真实误差 + 估计噪声；在 $M$ 个噪声里取最小值，会比典型值低约 $s\sqrt{2\log M}$（$s$ 是噪声的标准差）。只要 $M$ 大到 $s\sqrt{2\log M}$ 超过真实的差距，选择就主要由噪声决定 —— 纯噪声数据上 $M \ge 100$ 时，“只有截距”再也没被选中过。注意上界不需要独立性；相关的分数（比如共用同一个 $y$）实际的乐观偏差更小。

练习 6（Reading the optimizer-vs-objective test）. (a) Explain in your own words why "$J(\theta_B) \ge J(\theta_A)$" does not imply that A’s optimizer converged. (b) In demo 4 the costs are 10 (false positive) vs 1 (false negative). If A’s probabilities $P(\text{spam}\mid x)$ were perfectly calibrated, what threshold minimizes expected cost? Is changing the threshold a change of “objective” or of “optimizer”?

(a) 这个比较只涉及两个点 $\theta_A$、$\theta_B$。$J(\theta_B) \ge J(\theta_A)$ 只说明”在 $J$ 的排序下 A 不比 B 差”，再加上 $a(\theta_B) < a(\theta_A)$，就说明 $J$ 和 $a$ 的排序矛盾 —— 目标函数有问题。但 $\theta_A$ 离 $J$ 的最优点可能还很远（演示 4 的 50 步：0.3547 vs 0.1991），只要 B 的 $J$ 更差，比较就看不出来。(b) 判为垃圾邮件的期望代价是 $10\cdot P(\text{非垃圾}\mid x)$，判为正常的是 $1\cdot P(\text{垃圾}\mid x)$；当 $P(\text{垃圾}\mid x) > 10/11 \approx 0.91$ 时才判为垃圾。改阈值改的是决策规则，属于”目标”那一侧：训练出的概率模型不变，变的是你如何把概率变成行动（README 动手改一改 6 可以验证它能追上加权逻辑回归多少）。

代码相关的练习（“动手改一改”）见 code/lecture03/README.md。

10. 延伸阅读 · Further reading

• CS229 main notes（Ng & Ma），Ch. 8（bias–variance、double descent、sample complexity bounds）和 Ch. 9（regularization、implicit regularization、cross-validation、Bayesian statistics）：本讲的主要来源。
• CS229 learning theory notes（Ng）：Hoeffding、union bound、有限 / 无限假设类、VC 维的原版推导。
• Andrew Ng, “Advice for applying machine learning”（CS229 讲稿）以及 Machine Learning Yearning：§6 的诊断方法。
• Hastie, Tibshirani & Friedman, The Elements of Statistical Learning, Ch. 3（ridge / lasso）、Ch. 7（bias–variance、CV、one-standard-error rule、AIC/BIC）。
• Belkin, Hsu, Ma & Mandal, “Reconciling modern machine learning practice and the bias-variance trade-off”, PNAS 2019；Nakkiran et al., “Deep Double Descent”, ICLR 2020；Nakkiran, Venkat, Kakade & Ma, “Optimal Regularization Can Mitigate Double Descent”, ICLR 2021。
• Hastie, Montanari, Rosset & Tibshirani, “Surprises in High-Dimensional Ridgeless Least Squares Interpolation”, Annals of Statistics 2022：§2.3 公式的精确渐近版本。
• Zhang, Bengio, Hardt, Recht & Vinyals, “Understanding deep learning requires rethinking generalization”, ICLR 2017；Bartlett, Long, Lugosi & Tsigler, “Benign overfitting in linear regression”, PNAS 2020。
• Tibshirani, “Regression Shrinkage and Selection via the Lasso”, JRSS-B 1996；Friedman, Hastie & Tibshirani, “Regularization Paths for GLMs via Coordinate Descent” (glmnet), JSS 2010。
• Wager, Wang & Liang, “Dropout Training as Adaptive Regularization”, NeurIPS 2013：§4.4 的推广。
• Shalev-Shwartz & Ben-David, Understanding Machine Learning, Ch. 2–6：学习理论最清楚的教材。