Lecture 8 · 防御与可证明鲁棒性 · Defenses & Certified Robustness

对应 Maps to：Stanford CS329T Spring 2022 Week 7 —— adversarial defenses；attacks on attributions；defenses against attacks on attributions；fireside chat “How to certify robustness?”。阅读材料：Madry et al. 2018 “Towards Deep Learning Models Resistant to Adversarial Attacks”；Fromherz, Leino, Fredrikson, Parno & Păsăreanu 2021 “Fast Geometric Projections for Local Robustness Certification”；Dombrowski et al. 2019 “Explanations can be manipulated and geometry is to blame”；Ross & Doshi-Velez 2018 “Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients”；Slack et al. 2020 “Fooling LIME and SHAP”。作业：HW5 第 2 部分（Defense 1：用对抗样本做数据增强；Defense 2：走向 Madry 对抗训练；2022 课程总览把 defensive distillation 和 adversarial training 列为本模块的两种防御）。另参考 CMU 18-739 的两场讲座：Dimitris Tsipras “Robust Machine Learning: Progress, Challenges, Humans”，Nicholas Carlini “Towards Evaluating the Robustness of Neural Networks”。Fireside chat 没有公开的 slides，§6 的认证部分依据上面的论文和 CMU 组（Leino、Fredrikson 等，与课程团队关系密切）公开的论文写成。 配套代码 Code：code/lecture08/（PyTorch，CPU；5 个脚本 + 2 个工具模块，每个脚本 15–70 秒） 预计阅读时间：3 小时（含运行代码）

0. 本讲导航 · How to read this lecture

0.1 Learning goals

After this lecture you should be able to:

1. State what a defense must claim — a threat model plus robust accuracy under the strongest adaptive attack — and explain why an attack only gives an upper bound on robust accuracy while a certificate gives a lower bound.
2. 认出梯度掩蔽（obfuscated gradients）的三种形式（shattered、stochastic、vanishing / exploding），说出对应的自适应攻击（BPDA、EOT、直接攻击 logit），并用一张”红旗”清单检查一份防御评估。
3. Write adversarial training as a min–max problem, explain Danskin’s theorem in one sentence, and explain why training on a fixed set of adversarial examples (HW5’s Defense 1) fails while Madry’s training works — and what it costs.
4. 用 Tsipras et al. 的玩具分布解释为什么鲁棒性和准确率可能在原理上冲突，以及为什么鲁棒模型的梯度”更好看”。
5. Compare certification methods — exact verification (MILP / SMT / geometric projections), relaxations (IBP, CROWN), Lipschitz margins (GloRo), randomized smoothing — by what they prove, what they cost, and how loose they are; and connect all of them to Lecture 1’s “certify what you can, abstain on the rest”.
6. 说明解释本身为什么也是攻击面：梯度图的脆弱性（Ghorbani）、操纵与几何（Dombrowski）、骗过 LIME 的脚手架模型（Slack），以及哪些防御有用、哪些只是看起来有用。

0.2 先修与本讲范围 Prerequisites & scope

本讲是第 7 讲的另一半：第 7 讲问”怎样找到对抗样本”，本讲问”怎样让它们找不到、以及怎样证明它们不存在”。最后一节把同样的问题转向解释：如果解释是我们审计模型的工具（第 2–4 讲），那么它也需要鲁棒性。

0.3 本讲地图 Map

§1  一个防御要声明什么：threat model、上界与下界                 ← 代码 01 演示 1
§2  看起来鲁棒的防御：梯度掩蔽的三种形式                         ← 代码 01 演示 2
§3  自适应攻击与评估清单                                       ← 代码 01 演示 3
§4  对抗训练：min–max、Danskin、代价                           ← 代码 02 演示 2
§5  鲁棒性 vs 准确率；鲁棒模型的梯度                             ← 代码 02 演示 1、3
§6  可证明鲁棒性：精确验证、松弛、Lipschitz 与 GloRo               ← 代码 03
§7  Randomized smoothing                                     ← 代码 04
§8  对解释的攻击与防御                                         ← 代码 05
§9  全景：什么样的声明有根据；§10–14 小结、误区、练习、阅读、预告

0.4 Notation

§8.4 报告群体批准率差距时沿用第 1 讲的符号：group 0 − group 1。

0.5 关于数字 · About the numbers

所有数字都来自 code/lecture08/ 的实际输出（固定种子，同一台机器上重复运行逐字相同）。图像实验用第 3 讲的 16×16 横线 / 竖线世界，去掉了水印：信号是一根亮度 0.42–0.78、长 4–6 像素的短线，背景是 $\mathcal N(0, 0.3^2)$ 噪声。一个有用的尺度感：$L_\infty$ 扰动 $\epsilon \approx 0.3$ 就足以把短线抹掉、再画一根方向相反的，所以 $\epsilon$ 远小于 0.3 的扰动不应该改变正确答案。信贷实验用第 1 讲的世界和第 1 讲的 Model B。模型都很小（约 5k 参数），数字说明机制，不代表 ImageNet 上的量级。

1. 一个防御要声明什么 · What a defense must claim

1.1 Threat model first

A defense is a claim of the form “no adversary in class $\mathcal A$ can make the model wrong on more than X% of inputs”. Carlini et al.’s “On Evaluating Adversarial Robustness” (2019) — written by authors from both the attack and defense side — asks every paper to state three things before any number:

• Goals: untargeted (any wrong label) or targeted (a specific label)?
• Knowledge: white-box (weights and the defense mechanism), black-box (queries only), or something in between? The default for evaluating a defense is white-box including the defense itself — Kerckhoffs’s principle from cryptography: security must not depend on the secrecy of the mechanism.
• Capabilities: which perturbations — an $L_p$ ball of radius $\epsilon$, rotations, a sticker, a word substitution?

Carlini 在 CMU 的讲座里给”防御”下了一个朴素的定义：一个神经网络，它（1）在测试数据上准确，（2）能抵抗对抗样本。第二条必须相对于一个写清楚的 threat model 才有意义。第 1 讲 §5.3 的教训在这里是起点：随机噪声测的是平均情况，对手挑的是最坏情况。

1.2 Upper bounds and lower bounds

Robust accuracy is a statement about all points in the ball:

There are exactly two ways to say something about it (Carlini’s slide: “construct a proof of robustness, or demonstrate a constructive attack”):

• An attack finds a $\delta$ that flips the decision. Success proves non-robustness at that point; failure proves nothing. So the accuracy under any attack is an upper bound on RA. Stronger attacks give tighter (lower) upper bounds.
• A certificate proves that no $\delta$ in the ball flips the decision. So certified accuracy is a lower bound on RA.

真实的鲁棒准确率夹在两者之间。§2–§5 的所有数字都是上界；§6–§7 的数字是下界。一个防御论文报告的”PGD 下准确率 85%“，严格说只是”我们的这一个攻击没有把它降到 85% 以下”。

1.3 “The” robust accuracy is a minimum over attacks

代码 01 演示 1 对一个普通 CNN（干净准确率 0.920，500 张测试图）用不同强度的攻击：

Three readings:

1. Random noise that moves every pixel by a full ε = 0.2 costs 8 points; PGD at the same ε leaves 1–2%. Testing with noise would declare this model robust.
2. Stronger attacks give lower numbers (FGSM 0.552 → PGD-50×3 0.516 at ε = 0.05). Report the minimum over the attacks you tried.
3. PGD-20 and PGD-50 × 3 agree to within 0.016: the attack has roughly converged. Showing this convergence (more steps and restarts don’t lower the number) is part of a credible evaluation — otherwise readers cannot tell “robust model” from “under-run attack”. (PGD-10 is slightly below PGD-20 at ε = 0.2 — random-restart noise; differences of 0.01 on 500 images are within noise.)

2. 看起来鲁棒的防御：梯度掩蔽 · Obfuscated gradients

2.1 A short history of broken defenses

The years after Szegedy et al. (2014) and Goodfellow et al. (2015) produced dozens of defenses, and most were broken within months:

• Defensive distillation（Papernot, McDaniel, Wu, Jha & Swami 2016）报告把 MNIST 上 JSMA 攻击的成功率从约 95% 降到 0.5% 以下。Carlini & Wagner（2016 “Defensive Distillation is Not Robust to Adversarial Examples”；2017 “Towards Evaluating the Robustness of Neural Networks”）用一个在 logit 上定义的损失，几乎 100% 攻破了它。C&W 攻击就是为此而生的（第 7 讲）。
• ICLR 2018：Athalye, Carlini & Wagner（ICML 2018，“Obfuscated Gradients Give a False Sense of Security”）检查了 9 个被接收的白盒防御，发现 7 个依赖于 obfuscated gradients；在各自论文声称的 threat model 下，他们完全攻破了 6 个、部分攻破了 1 个。
• 2020：Tramèr, Carlini, Brendel & Madry（NeurIPS 2020，“On Adaptive Attacks to Adversarial Example Defenses”）又攻破了 13 个发表在 ICLR、ICML、NeurIPS 上的防御 —— 这些论文都声称做了自适应评估。

The common pattern: the defense didn’t make the model robust; it made the gradient useless, so gradient-based attacks failed. Athalye et al. call this obfuscated gradients and identify three kinds.

2.2 Vanishing gradients: how defensive distillation “worked”

The recipe (implemented in robust_tools.distill):

1. Train a teacher with logits divided by a temperature $T$: loss $= \text{BCE}(\sigma(z/T), y)$.
2. Use its soft labels $\sigma(z_{\text{teacher}}/T)$ to train a student, also at temperature $T$.
3. Deploy the student at $T = 1$.

Because training divided the logits by $T$, the student learns logits about $T$ times larger than a normal network. In our run with $T = 100$, the deployed student’s median $|z|$ is 319.4, and 96% of test images have $|z| > 17$. In float32, $\sigma(z) = 1/(1 + e^{-z})$ evaluates to exactly 1.0 once $z \gtrsim 17$ (the $e^{-z}$ term is lost when added to 1) and to exactly 0 once $z \lesssim -104$ (underflow) —— so an attacker who computes the cross-entropy on the output probability gets $\partial p/\partial z = p(1-p) = 0$ exactly; §3.2 counts 91% of test images with an exactly-zero gradient. The gradient vanishes; PGD doesn’t move. Nothing about the decision boundary changed.

修复方法简单得令人尴尬：直接攻击 logit（C&W 的 margin 损失 $-s\cdot z$），或者把 logit 除以 $T$ 再攻击。代码 01 演示 2 的数字（500 张图，PGD-20）：

naive = the attack a defense author would most likely run (standard PGD on the defended model; for distillation, cross-entropy on probabilities). adaptive = the countermeasure from §2.3. transfer = adversarial examples crafted on the undefended model and replayed against the defense.

Distillation: naive 0.836 at ε = 0.1, logit attack 0.114 —— no better than no defense (0.142).

2.3 Three kinds of obfuscation, three countermeasures

Quantization（shattered）：网络在量化后的图像上训练，部署时先 round 到 0.25 的网格。torch.round 的导数是 0，天真的 PGD 在 ε = 0.1 时得到 0.910 —— 和干净准确率（0.914）一样，ε = 0.05 时甚至是 0.928，比干净还高（随机起点碰巧把几张图”量化”对了）。BPDA 的写法只有一行：q = x + (round(x) − x).detach()，前向的值是 round(x)，对 x 的梯度是 1。换上 BPDA，0.176。

Randomization（stochastic）是三者中最弱的掩蔽：单次前向的梯度已经相当好用（ε = 0.1：0.366），8 次平均的 EOT 再压到 0.270。随机性本身不是防御 —— 但把随机性和一个定理结合起来，就是 §7 的 randomized smoothing。

2.4 Masking is not robustness

The clearest symptom in the table: for quantization and distillation, the black-box transfer attack beats the white-box attack by a wide margin (quantization at ε = 0.1: transfer 0.248 vs white-box 0.910). An attacker who knows less succeeds more. That can only mean the white-box attack is broken, not that the model is robust. The decision boundary is where it always was; the defense merely hid the signpost pointing to it.

这也解释了为什么”迁移攻击”是评估防御的必备项：它不依赖被防御模型的梯度，所以不受梯度掩蔽影响。

3. 自适应攻击与评估清单 · Adaptive attacks and evaluation checklists

3.1 The adaptive-attack principle

An adaptive attack is one designed with full knowledge of the defense, targeting whatever the defense does. Tramèr et al. (2020) distilled their 13 case studies into advice that sounds simple but is routinely ignored:

• There is no universal “strong attack” to run. Understand the defense’s mechanism, then design the loss and the gradient estimator for it.
• Prefer simple attacks that you can verify are working (loss going down, success at large ε) over complex ones.
• Make the attack’s loss consistent with the goal: if the defense outputs probabilities that saturate, attack logits; if it has a detector, attack the combined system.

对防御作者来说，自适应攻击是自己的责任：在论文里给出针对自己机制的最强攻击，而不是只跑一个现成的 PGD。

3.2 Red flags you can compute

Athalye et al. (2018) and Carlini et al. (2019) list symptoms that indicate a broken evaluation. 代码 01 演示 3 把其中三条变成自动检查（ε = 0.1；括号里是 ε = 1 时自适应攻击后的准确率）：

• zero-grad：攻击损失对输入的梯度恰好全为 0 的测试图比例。正常网络几乎不会出现；量化 100%、distillation 91%。
• (b) 的逻辑：ε = 1 远超”把短线抹掉”所需的 0.3，任何正常工作的攻击都应该接近 100% 成功。
• The undefended model raises flag (b) too. With cross-entropy loss at ε = 1, 6.2% of images survive. The cause is the same as distillation’s, in miniature: the large random start pushes some images to $|z| > 17$, where $\sigma(z) = 1.0$ in float32 and the cross-entropy gradient is exactly zero (even 2.0% of clean images have exactly zero gradient). Switching to the margin loss gives 0.000. A red flag is about the evaluation, not the defense — this is precisely what it should catch, and it is one reason C&W and later AutoAttack use margin-type losses on logits.
• No flags ≠ robust. The randomized defense passes all three checks, yet its robust accuracy is only 0.27 at ε = 0.1. Red flags detect broken evaluations; only stronger attacks (or proofs) establish robustness.

3.3 A practical checklist

综合 Carlini et al.（2019）的清单，报告一个经验性防御时至少应该做到：

1. 写清 threat model（目标、知识、能力），并在这个 threat model 下评估。
2. 用自适应攻击：针对机制设计损失和梯度估计（BPDA、EOT、logit 损失）。
3. 证明攻击收敛：增加步数、随机重启，数字不再下降；画出 ε 与成功率的曲线，ε 足够大时应接近 100%。
4. 同时跑迁移（黑盒）攻击和无梯度攻击（基于查询的攻击）；它们不应比白盒更强。
5. 检查梯度：恰好为 0 的比例、梯度攻击 vs 随机噪声。
6. 使用标准化的攻击集合 —— AutoAttack（Croce & Hein 2020：四个互补的无参数攻击）—— 并和 RobustBench 排行榜上的结果对照；公开代码和模型。

Even all six do not make an empirical claim a proof. They make it falsifiable — which is what turned the field around after 2018.

4. 对抗训练 · Adversarial training

4.1 Data augmentation: random, static, or adaptive?

The most natural defense is to show the model what it will face. HW5 part 2 builds it up in steps, and 代码 02 演示 2 reproduces them on the bars world (1000 test images; training ε = 0.1; PGD-20 against each model itself):

fixed adv = one batch of adversarial examples generated once against the standard model (ε = 0.1): “train” are the ones the static model was trained on, “test” the same recipe on test images. cost = forward–backward passes per training example (10 epochs).

• 高斯噪声增强买到了一点（ε = 0.1：0.141 → 0.321）。这个任务的背景本来就是噪声，所以噪声增强比通常更有用；但仍然远不如对抗训练。随机扰动是平均情况，不是最坏情况。
• 静态对抗增强是最有教育意义的一行。它对训练时见过的那批对抗样本几乎全对（0.998），对同样方法生成的新样本也是 0.961 —— 可是对它自己做一次新的白盒攻击，只剩 0.012，比普通模型还差。它学会的是”认出那种扰动的样子”，而不是”对整个 ε 球都不变”。新模型有新的最坏情况。HW5 给出的修正是：在训练过程中、针对当前的模型生成对抗样本。

4.2 The min–max formulation and Danskin’s theorem

Madry et al. (2018) make this precise. Standard training minimizes the expected loss; adversarially robust training minimizes the expected worst-case loss:

两层优化：内层是一个攻击（找最坏的 δ），外层是训练（在最坏的点上降低损失）。外层要对”一个最大值”求梯度 —— 怎么求？Tsipras 讲座里的 Danskin 定理给出答案（在适当的正则条件下）：

**在最大值点上求梯度，就是最大值的梯度。**所以算法很简单：每个小批量先用 PGD 找到 $\delta^\star$，再在 $x + \delta^\star$ 上做一步普通的梯度下降。两个关键条件（Tsipras 的讲座称之为两个 key ingredients）：

1. Reliable attacks. Danskin needs the true maximizer. The loss is non-convex in δ, but Madry et al. found that PGD from random starts reaches consistently high, similar loss values — good enough in practice. FGSM-based adversarial training (Goodfellow et al. 2015) uses a crude one-step maximizer and is prone to catastrophic overfitting: the model learns to defeat FGSM specifically while PGD robustness collapses (Wong, Rice & Kolter 2020 “Fast is better than free” fixed it with random initialization and careful step sizes). Static augmentation (§4.1) is the extreme case: a maximizer for the wrong θ.
2. Capacity. Robust classifiers may need more capacity; weak models can fail to train at all. Our first attempt at PGD-AT with the full ε = 0.1 from the first step collapsed to a constant classifier (accuracy 0.464). Ramping ε up from 0 over the first epochs fixed it. (Following the capacity point, our CNN already has twice the channels of Lecture 3’s.)

4.3 What it buys, and what it costs

PGD-AT raises robust accuracy at ε = 0.1 from 0.141 to 0.628; clean accuracy drops from 0.908 to 0.892; training costs 6× (5 PGD steps + 1 update per example). On real benchmarks the cost is the main practical obstacle, which motivated cheaper variants (Shafahi et al. 2019 “Adversarial Training for Free!”; Wong et al. 2020).

Re-checking a good defense with stronger attacks (ε = 0.1, first 500 test images):

The numbers go down by 0.01–0.04 with a stronger attack: PGD-20 had not fully converged on the adversarially trained model, and the number to report is the lower one. But nothing like §3’s red flags appears — transfer is no stronger than white-box, the loss choice barely matters. Adversarial training was the defense that survived Athalye et al.’s 2018 study, and it remains the basis of most entries on RobustBench. For scale, Tsipras’s talk lists robust accuracies of about 89% on MNIST ($L_\infty$, ε = 0.3), 53% on CIFAR-10 (ε = 8/255) and 33% on ImageNet (ε = 4/255) — far below clean accuracy on the two harder datasets.

4.4 TRADES, robust overfitting, and sample complexity

• TRADES（Zhang et al. 2019，NeurIPS 2018 Adversarial Vision Challenge 的冠军方法）把目标拆成两项：$\ell(f(x), y) + \beta\cdot \mathrm{KL}\big(p(x)\,\Vert\, p(x')\big)$，$x'$ 是让 KL 最大的点。第一项管干净准确率，第二项管”x 附近预测不要变”，β 控制权衡。我们的单次运行里 TRADES（β = 6）的干净和鲁棒准确率都略低于 PGD-AT（0.866 / 0.582 vs 0.892 / 0.628），没有落在”更准、稍不鲁棒”的预期位置；在这么小的模型和训练预算下，一次运行不足以比较两种方法。
• Robust overfitting（Rice, Wong & Kolter 2020）：对抗训练时，训练集上的鲁棒准确率一直上升，测试集上的鲁棒准确率在学习率下降后不久就开始下降 —— 标准训练里很少见的现象。早停（按验证集上的鲁棒准确率）是最有效的对策之一。
• Sample complexity（Schmidt et al. 2018）：存在这样的分布，一个样本就足以学到好的标准分类器，鲁棒分类器却需要 $\Omega(\sqrt d)$ 个样本。Tsipras 的讲座里，对抗训练的鲁棒准确率在训练集和测试集之间相差超过 50 个百分点 —— 标准训练里不会这样。鲁棒泛化需要更多数据。

5. 鲁棒性 vs 准确率，以及鲁棒模型的梯度 · Robustness vs accuracy, and gradients that look right

5.1 A distribution where you cannot have both

Tsipras, Santurkar, Engstrom, Turner & Madry (2019) constructed a simple distribution that makes the conflict exact. 代码 02 演示 1 的版本：

• $y \in \{-1, +1\}$ equally likely.
• One robust feature $x_1 = y$ with probability $p = 0.9$, else $-y$. Alone it gives 90% accuracy, and since it is ±1, a small perturbation cannot flip its sign.
• $d = 100$ non-robust features $x_j \sim \mathcal N(\eta y, 1)$ with $\eta = 2/\sqrt d = 0.2$. Each alone is nearly useless (accuracy $\Phi(0.2) \approx 0.58$), but their average has signal-to-noise ratio $\eta\sqrt d = 2$, so together they give about 98%.
• An $L_\infty$ adversary with ε = 2η = 0.4 shifts every weak feature by $-2\eta y$, turning $\mathcal N(\eta y, 1)$ into $\mathcal N(-\eta y, 1)$: the whole group defects at once.

For a linear model the inner maximization is exact — the worst case lowers the margin by exactly $\epsilon\lVert w\rVert_1$ (Lecture 1 §5.3) — so adversarial training is just logistic regression on the margin $y(w^\top x + b) - \epsilon\lVert w\rVert_1$. The $\lVert w\rVert_1$ term acts like an L1 penalty: it punishes many small weights and favors a few large ones. Lecture 1 §7.3’s certificate $|z| > \epsilon\lVert\theta\rVert_1$ has become a training objective.

Standard training uses the weak features (they are predictive) and reaches 0.983 clean but 0.054 robust. As the training ε grows, weight moves from the weak features to $x_1$, and the model ends at $p \approx 0.9$ on both. The transition is not smooth: up to ε = 0.1 the model still mostly relies on the weak features, at ε = 0.2 it switches. On this distribution you get the weak features’ accuracy or robustness, not both.

这和第 7 讲的 “Adversarial examples are not bugs, they are features”（Ilyas et al. 2019）是同一枚硬币的两面：非鲁棒特征是真实的、有预测力的特征，标准训练没有理由不用它们；而要鲁棒，就必须主动放弃它们。Tsipras 讲座的总结：“Standard training: use all the features to maximize accuracy. Adversarial training: use only strong features (lower accuracy).”

在图像世界里也看得到这个取舍的影子：PGD-AT 的干净准确率比普通模型低 0.016，TRADES 低 0.042。在真实的 CIFAR-10 上差距更大：Madry et al. 的对抗训练模型干净准确率约 87%，同样结构的标准训练模型约 95%。

5.2 Robust models look at the right pixels

A side effect that surprised the field: the input gradients of robust models look like the object. Tsipras et al. (2019) and Santurkar et al. (2019) call these perceptually aligned gradients; Tsipras’s talk shows that maximizing a class score of a robust model produces recognizable features (“class maximization introduces salient features”), enough for image generation and inpainting with a single robust classifier.

代码 02 演示 3 用第 3 讲 §4.1 的同一个指标量化它：输入梯度 $|\nabla_x z|$ 有多大比例落在短线的 4–6 个像素上（约占面积的 2%）：

One test image (a horizontal bar in row 6; left: the image, middle: standard model’s $|\nabla_x z|$, right: PGD-AT’s; each normalized by its own maximum):

       ::..      ..  --        ::     |                                      |                                   
     ..==..  --    --    ....         |                                      |                                   
   ......          ::==**..::         |                   ......  ..         |                                   
     ==      ::  ..  ++..==  ==  ..   |                       ....  ..       |                                   
   ..      ::    ::  ::    **----     |                   ::      ....  ..   |                                   
   ..    ::..--..  --  ==        ++   |                   ++::....::  ....   |                   ..              
   ..==..==  ::..  %%  ..--**++**     |                   ##::##%%@@%%  ..   |                   ::  ::--@@**::  
       ::      ==++==::        ++..   |         ..    ..  ##....::--..##..   |                   ::          --  
   ====    --      ..  ==  --  ++     |             ..  ..==..    ..  ##..   |                   ..          --  
         --..--..  ::    ....  ..     |                 ..::    ....  --..   |                                   
   ==        ::  ##..      ==  ..     |             ..    ....  ....         |                                   
   ::    **      ::    --        ..   |                 ..    ..             |                                   
               ==  ..    ::      ..   |                                      |                                   
     ::==      ==  ##      ..++       |               ..::::                 |                                   
   ==  ..  @@==      --..      ==     |         ..                           |                                   
   --    ::::==  ..        --==  **   |                                      |                                   

Three observations:

• 对抗训练的模型把梯度集中在短线上（0.694），普通模型的梯度散在背景噪声里 —— 就是第 3 讲里那种”嘈杂”的 saliency 图。为什么？对抗训练惩罚”沿着很多微弱方向都敏感”的模型（§5.1 的 $\lVert w\rVert_1$），逼它只依赖”要改变它就得付出很大扰动”的特征 —— 在这个世界里就是短线本身。
• SmoothGrad did not help here（0.174 < 0.216）：σ = 0.3 的噪声把梯度”抹”到了短线周围更大的范围。平滑解释改变的是我们怎么看模型；对抗训练改变的是模型依赖什么。
• The static-augmentation model is worst (0.092) and puts the most mass on the top-left corner (0.034): it learned to look for the perturbation, not the bar.

Regularizing input gradients. Ross & Doshi-Velez (2018) go the other way: penalize $\lVert\nabla_x \ell\rVert^2$ during training (“double backpropagation”, Drucker & LeCun 1992). Small input gradients mean a flatter loss around each training point, which they found improves both robustness to some attacks and the interpretability of gradients, at lower cost than adversarial training. 我们在编写本讲时做过一个小原型（不在最终代码里）：在这个世界上，λ = 1 让干净准确率掉到 0.838，却没有带来明显的鲁棒性（ε = 0.1 的 PGD 下 0.165，普通模型约 0.14）；λ = 10 让模型塌缩成常数 —— 这类正则对强度很敏感，见 README”关于实验结果”和”动手改一改”。

这一节是第 1 讲 §7.4 那张”跨透镜相互作用”表里 “Robustness ↔ explainability” 一行的具体内容：鲁棒性和可解释性在这里是同一件事的两个侧面。

6. 可证明鲁棒性 · Certified robustness

6.1 From Lecture 1’s one-line certificate to deep networks

Lecture 1 §7.3 proved, for a linear model $z = \theta_0 + \theta^\top x$: if $|z(x)| > \epsilon\lVert\theta\rVert_1$, no $L_\infty$ perturbation of size ε can flip the decision. The general form uses the dual norm: for an $L_p$ ball, the worst-case change of $\theta^\top x$ is $\epsilon\lVert\theta\rVert_*$ (Hölder’s inequality), so

代码 03 演示 1 用的就是第 1 讲的 Model B 和同一批 4000 个部署申请人（‖θ‖₁ = 3.318，‖θ‖₂ = 1.570）：

• 前两列就是第 1 讲 §7.3 的表（coverage 0.930 / 0.866 / 0.652 / 0.368）。
• 同一个 ε，$L_2$ 下能证明的人多得多（ε = 0.25：0.841 vs 0.652），因为 $L_2$ 球是 $L_\infty$ 球的一个子集。威胁模型的选择直接决定”多少人可以被证明” —— 这是必须写进报告的假设，不是技术细节。
• mismatch = 0: for a linear model, PGD started at δ = 0 walks straight to the exact worst case (the gradient direction never changes), so attack and certificate agree person by person. Upper bound = lower bound; robustness is computed, not estimated. (In an earlier run where PGD started from a random point inside the $L_2$ ball, the projection step left a sideways component, PGD fell slightly short of the worst case, and up to 15 applicants disagreed — even on a linear model an attack can be a little weak.)

For deep networks there is no closed form. Four families of methods fill the gap.

6.2 Exact verification, and Fast Geometric Projections

Exact (complete) verifiers answer the question “is there an adversarial example in the ball?” with yes or no. Reluplex (Katz et al. 2017) extends the simplex method to handle ReLU constraints (an SMT approach); Tjeng, Xiao & Tedrake (2019) encode the network as a mixed-integer linear program (MILP), one binary variable per uncertain ReLU. Katz et al. also proved the problem NP-complete for ReLU networks. Tsipras’s slide summarizes the state of the field: exact verification is “accurate but intractable”; relaxations are efficient but “bounds might be too loose”; “accurate and efficient verification largely open”.

Fast Geometric Projections（FGP；Fromherz, Leino, Fredrikson, Parno & Păsăreanu，ICLR 2021，本周的阅读材料）利用 ReLU 网络的几何结构，不调用任何求解器：

• A ReLU network is piecewise linear. Fix which ReLUs are on and off (an activation pattern) and the whole network becomes an affine function; the inputs sharing a pattern form a convex polytope, an activation region. Inside a region the decision boundary is a single hyperplane.
• Search the regions around $x$ breadth-first. In each region: (i) compute the distance from $x$ to the region’s decision hyperplane; if it is < ε, project $x$ onto it — if the projection lies inside the region and flips the label, that’s a real adversarial example (“not robust”); otherwise mark “unknown”; (ii) for each neuron, compute the distance from $x$ to its hyperplane; if < ε, the neighboring region (that neuron flipped) might intersect the ball, so enqueue it.
• Why it is sound: any region that intersects the ball can be reached from $x$ along a straight segment inside the ball, and every boundary the segment crosses is within ε of $x$. Using distance to the hyperplane rather than to the actual facet over-approximates — FGP may explore extra regions, but never misses one. If no explored region has a decision boundary within ε, the point is certified.
• It is incomplete: “unknown” when a boundary is near but the projection falls outside the region. The paper reports it is far faster than complete verifiers and especially strong for $L_2$.

certify_tools.fgp_certify 是一个简化实现（numpy，逐点 BFS，不做 GPU 并行）。

6.3 Relaxations: IBP and friends

Instead of searching exactly, relaxations over-approximate the set of reachable activations by something simple and propagate it layer by layer.

Interval bound propagation（IBP；Gowal et al. 2018；思想来自 abstract interpretation，Gehr et al. 2018 的 AI²）：每个单元一个区间 [lo, hi]。第一层对 $L_2$ 球是精确的（$w^\top x \in w^\top x_0 \pm \epsilon\lVert w\rVert_2$）；ReLU 把区间截断到 ≥ 0；之后的线性层用”中心 ± 半径”：$(W\mu + b) \pm |W| r$。松在哪里？每一层都假设各单元能独立地取到区间端点，忽略了它们来自同一个输入 —— 越深越松。更紧的方法用线性上下界代替区间：convex outer polytope（Wong & Kolter 2018）、Fast-Lin / CROWN（Weng et al. 2018；Zhang et al. 2018），以及把它们和分支定界结合的 α,β-CROWN。这些界也能放进训练目标（IBP training、CROWN-IBP），让模型变得”容易被证明”。

6.4 Four bounds on the same number

代码 03 演示 2：第 1 讲世界上的一个 5 → 16 → 16 → 1 ReLU 网络（干净准确率 0.746，前 500 个部署申请人，标准化空间的 $L_2$ 球）：

（PGD = $L_2$ PGD-50 × 2 次重启，margin 损失；FGP / IBP / Lipschitz = 证书成立且分对的比例；budget = 搜索超过 1000 个区域后放弃；violations = PGD 攻破了、FGP 却说 robust 的点 —— 必须是 0；PGD missed = FGP 找到了真的对抗样本、PGD 却没找到。）

• 在这个网络上，顺序是 PGD ≥ FGP ≥ IBP ≥ Lipschitz：一个上界，三个越来越松的下界（PGD 一定在最上面；三个下界之间的顺序不是定理 —— §6.5 的 GloRo 网络上 Lipschitz 反而最高）。violations 处处为 0 —— 证书是可靠的。
• ε = 0.05 时 PGD 和 FGP 只差 0.002：真实的鲁棒准确率被夹住了。ε = 0.3 时缝隙变成 0.622 vs 0.486，因为 FGP 用完了预算或遇到 unknown。
• ε = 0.3 时，FGP 找到了 2 个 PGD-50 × 2 没找到的真对抗样本。攻击确实只是上界。
• 全局 Lipschitz 证书最松：$\prod_l \lVert W_l\rVert_2 = 8.56$ 假设每一层都把扰动往最坏的方向放大，而这个网络在任何局部都远没有这么陡（线性模型的 $\lVert\theta\rVert_2$ 只有 1.570）。

精确验证的代价：把网络宽度加倍（5 → 32 → 32 → 1），前 200 个申请人，同样 1000 个区域的预算：

这就是 NP-完全性的样子：要搜索的区域数随 ε 和宽度指数增长。精确的方法不扩展，能扩展的方法（IBP、Lipschitz）不精确。

6.5 Lipschitz margins and GloRo: train for the certificate

The Lipschitz certificate generalizes Lecture 1’s linear one directly: if the logit is $L$-Lipschitz in $L_2$, then $|z(x)| > \epsilon L$ certifies radius ε. It costs one forward pass. The problem is that for a normally trained network the cheap upper bound $L \le \prod_l \lVert W_l\rVert_2$ is hopelessly loose. Tsuzuku, Sato & Sugiyama (2018, “Lipschitz-Margin Training”) proposed training so that the margin exceeds εL; GloRo (Leino, Wang & Fredrikson, ICML 2021, “Globally-Robust Neural Networks”) turns it into an extra output class.

GloRo adds a class ⊥ (“I won’t commit”) whose logit is

⊥ 赢不了 ⇔ 其他每个类即使”最坏地”往上涨 $\epsilon K_{ij}$ 也追不上 $j$ ⇔ 预测在半径 ε 内可证明不变。训练时对增加了 ⊥ 的 logit 用普通交叉熵（真实标签永远不是 ⊥），模型被逼着把 margin 做大、把 Lipschitz 上界做小 —— 让最便宜的证书变得够用。二分类时把两个类的 logit 写成 $(0, z)$，条件就化成 $|z| > \epsilon K$：第 1 讲的 certify-or-abstain，只是弃权变成了模型的一个输出，而且写进了训练目标。论文里的名字 “globally robust” 指的是：对任意两个距离不超过 ε 的输入，模型要么给出相同的类别，要么其中一个是 ⊥。

代码 03 演示 3（同样的网络结构，GloRo 训练，ε_train = 0.3）：

• Lipschitz 上界从 8.56 降到 1.615 —— 几乎和线性模型（1.570）一样平缓。
• 最便宜的证书现在在每个 ε 上都等于 PGD 上界：真实的鲁棒准确率被精确地确定了，每个输入只要一次前向。这才是部署时能对每个请求都算一遍的证书。
• FGP 在 ε = 0.3 时反而不如 Lipschitz（0.620 vs 0.646）：2.6% 的点撞上了搜索预算。实践中证书常常串起来用 —— 先用便宜的，证不出来再用贵的。
• 代价：干净准确率 0.746 → 0.722；ε = 0.3 时 13.4% 的申请人得到 ⊥（弃权、转人工）。
• GloRo 在 ε = 0.3 的证书下界（0.646）高于普通网络的攻击上界（0.622）。为证书而训练，不只是让模型更容易被验证，也让它真的更鲁棒。

6.6 Comparing certificate families

7. Randomized smoothing

7.1 The idea

所有上面的方法都要”打开”网络（逐层传播、搜索区域、计算谱范数）。Cohen, Rosenfeld & Kolter（ICML 2019，“Certified Adversarial Robustness via Randomized Smoothing”；更早的 PixelDP 见 Lecuyer et al. 2019）换了一个问题：不证明原来的分类器 $f$，而是证明它的平滑版本

Theorem (binary case). If class A satisfies $P[f(x + \delta) = A] \ge \underline{p_A} > 1/2$, then $g$ predicts A everywhere in the $L_2$ ball of radius

(The general statement is $R = \frac{\sigma}{2}\big(\Phi^{-1}(\underline{p_A}) - \Phi^{-1}(\overline{p_B})\big)$; with two classes $\overline{p_B} = 1 - \underline{p_A}$.) Intuition: moving the input by $R$ moves the center of the Gaussian by $R$; if A holds a large enough share of the Gaussian’s mass, a small shift cannot hand the majority to B. The worst case is when $f$‘s A-region is a half-space, which is where $\Phi^{-1}$ comes from (a Neyman–Pearson argument). The theorem needs nothing about $f$ — it can be any network, even a black box.

7.2 Monte Carlo, Clopper–Pearson, abstain

$p_A$ 只能抽样估计，所以 CERTIFY 算法（certify_tools.certify_smoothed）是：

1. 用 $n_0$ 个噪声样本选出最可能的类别 A；
2. 用另外 $n$ 个新样本数 A 出现了 $k$ 次，取单侧 Clopper–Pearson 下界 $\underline{p_A}$（置信度 1 − α）；
3. $\underline{p_A} < 1/2$ ⇒ 弃权；否则返回 (A, $\sigma\Phi^{-1}(\underline{p_A})$)。

为什么选和估计要用两批样本？用同一批样本先挑”赢家”再估计它的概率，估计会偏高 —— 第 1 讲 §8.2 的赢家诅咒。代码 04 演示 1（α = 0.001）：

Even if all samples agree, $\underline{p_A} = \alpha^{1/n}$ caps the radius: $2.46\sigma$ with $n = 1000$, $3.20\sigma$ with $n = 10{,}000$. With unlimited samples $k/n = 0.6$ would give $\Phi^{-1}(0.6) = 0.253$; the finite-sample price is visible (0.128 at $n = 1000$). Cohen et al. used $n = 100{,}000$ on ImageNet — the dominant deployment cost of smoothing is these forward passes.

7.3 Results on the bars world

代码 04 演示 2（前 250 张测试图，$n_0 = 64$，$n = 1000$；certified acc @ r = 预测正确且半径 ≥ r 的比例，弃权算错）：

• **σ 的权衡。**σ 越大，能证明的半径越大（上限 $2.46\sigma$：σ = 0.25 时 0.62，σ = 0.5 时 1.23；r = 0.75 只有 σ = 0.5 证出了一点），但基分类器在噪声下越不准，小半径上的认证准确率越低（r = 0：约 0.83 vs 0.68–0.70）。
• 噪声训练没有像预想的那样起作用。Cohen et al. 用同样 σ 的高斯噪声训练基分类器，在真实数据集上这一步非常重要。在这里，σ = 0.25 时两个基分类器几乎一样 —— 这个世界的背景本来就是 $\mathcal N(0, 0.3^2)$ 噪声，普通训练就是带噪声的训练。σ = 0.5 时，噪声训练的模型在噪声下更常对（0.672 vs 0.636）、r = 0 时更好，但弃权更多（0.196 vs 0.116），r ≥ 0.25 时证出来的更少 —— 它更常对，却不够坚决，$\underline{p_A}$ 常落在 1/2 附近。我们保留了这个结果，没有为了故事去调参（见 README）。
• **半径都很小。**r = 0.5 时最好的认证准确率只有 0.144，而抹掉短线再画一根需要 $L_2 \approx 1.9$。这和文献一致：ImageNet 上 smoothing 的证书也只覆盖人眼远远察觉不到的扰动。

7.4 Is the certificate sound? Certified vs empirical

A certificate should survive any attack. 代码 04 演示 3 攻击平滑分类器本身：对软平滑分类器 $G(x) = \mathbb E[\sigma(f(x + \delta))]$ 做 $L_2$ PGD，每步用 16 个噪声样本估计梯度（EOT，Salman et al. 2019 “SmoothAdv” 的思路），然后在攻击后的点上重新运行 CERTIFY（σ = 0.25 的噪声训练模型，前 120 张图）：

• violations = 0：被认证的图一张都没有被攻破。对比 §2：被掩蔽的防御一换攻击，数字就崩了；证书的数字没有。（当然，一次攻击失败不是证明 —— 证明是定理本身，攻击只是检查实现有没有写错。）
• 证书是保守的：经验鲁棒准确率高于认证准确率（r = 0.5：0.275 vs 0.125）—— 很多没被认证的图其实攻不破，只是它们的 $\underline{p_A}$ 不够大，证明不出来。
• Smoothing 买到的主要不是更多的鲁棒性 —— 没有平滑的普通模型在 PGD-$L_2$ 下的经验数字差不多（0.617、0.258）—— 而是可以被证明的鲁棒性，对任何网络结构都适用。代价是每次预测约 1000 次前向、一部分弃权，以及一个这样的保证：(i) 概率性的（以至多 α 的概率出错）；(ii) 关于 $g$ 而不是 $f$ —— 部署的必须是 $g$；(iii) 只对 $L_2$ 自然：半径 ε 的 $L_\infty$ 球包含 $L_2$ 距离为 $\epsilon\sqrt d$ 的点（我们的 256 个像素是 16ε），所以高斯平滑给出的 $L_\infty$ 证书在高维下很弱。

“证明能证的、其余弃权”在这里又出现了一次：弃权率是 smoothing 必须报告的数字之一。

8. 对解释的攻击 · Attacks on explanations

8.1 Why explanations are an attack surface

第 2–4 讲把解释当作审计模型的工具；第 1 讲 §4.1 提醒过：一个漂亮的解释可能增加信任、同时降低可信度。解释也是一个函数的输出：$x \mapsto h(x)$（梯度图、IG、LIME 系数、SHAP 值）。所以本讲前七节的问题原样回来了：它在扰动下稳定吗？能被刻意操纵吗？对手是谁？

Two adversaries matter. (1) Someone perturbs the input so the explanation changes while the prediction doesn’t — making an audit point at the wrong evidence. (2) The model owner builds a model whose explanations hide what it does — the auditor is the target. §8.2–8.3 are the first kind, §8.4 the second.

8.2 Fragility of gradient maps (Ghorbani et al. 2019)

Ghorbani, Abid & Zou (AAAI 2019, “Interpretation of Neural Networks is Fragile”) showed that small perturbations that keep the predicted label can drastically change saliency, DeepLIFT and IG maps. Their main metric is top-k intersection: what fraction of the k most important features remain the most important after the perturbation.

代码 05 演示 1：普通 ReLU CNN，100 张测试图。攻击最小化”原解释的 top-10 像素在新解释里占的比例” + γ·(Δz)²（把 logit 钉住）。ReLU 网络的梯度是分段常数，它对输入的导数几乎处处为 0，所以攻击用 softplus_β（β = 10）的副本求二阶导，再在真正的 ReLU 网络上检验：

ε = 0.1 时，最重要的 10 个像素平均只剩 3.3 个，而没有一个预测改变，logit 平均只动了 0.002（这些图的平均 |logit| 是 5.67）。同样大小的随机噪声只换掉约四分之一。注意即使是随机噪声也会改变 top-10：输入每跨过一个激活区域的边界，ReLU 网络的梯度就跳变一次 —— 这和 §6.2 里 FGP 搜索的区域是同一个几何对象。对第 3 讲的审计流程来说，这意味着一张解释图只描述这一个点附近的模型。

8.3 Manipulation and geometry (Dombrowski et al. 2019)

Dombrowski, Alber, Anders, Ackermann, Müller & Kessel (NeurIPS 2019, “Explanations can be manipulated and geometry is to blame”) go further: a perturbation can make the explanation match an arbitrary target map while the output stays nearly constant. Their explanation of why: for a gradient-type explanation, how much it can change under a small input change is controlled by the curvature of the hypersurfaces of constant network output — ReLU networks have flat pieces joined by sharp kinks, i.e. very large curvature at the kinks. Their proposed fix, β-smoothing: compute the explanation with ReLU replaced by softplus_β, $\log(1 + e^{\beta h})/\beta$. Smaller β bounds the curvature; they relate the result to SmoothGrad-like averaging.

代码 05 演示 2，ε = 0.1。(a) 目标攻击：给每张图配一张另一个类别的图，用它的解释做目标（“竖线的图，解释却像一根横线”）。攻击后的解释和目标图的相关从 0.091 升到 0.281，和原解释的相关降到 0.698，预测翻转 1%。在这么小的预算和网络里，解释能被明显地拉向目标，但拉不到”一模一样”；Dombrowski et al. 在大网络和自然图像上做到了几乎任意的目标图 —— 输入维数越高，对手的自由度越大。

(b) β-smoothing，同样的权重：

（top-10 overlap，越高越稳定。transfer = 把针对 ReLU 梯度的攻击原样拿来；adaptive = 直接攻击平滑后的解释本身。random 一列和上表的 0.733 不同，是因为抽了不同的随机噪声。）

• 又是自适应攻击的教训。面对迁移过来的攻击，β = 3 看起来比 ReLU 梯度稳得多（0.524 vs 0.330）；直接攻击平滑后的解释，它就掉到 0.377。解释的防御必须用针对那个解释的攻击来评估 —— 正是 §3 的原则。
• 平滑确实有用，诚实地量：自适应攻击下 β = 1 仍保留 0.491。曲率更小，更难操纵 —— 和 Dombrowski 的理论一致。
• 代价是忠实度。β = 1 的解释和真正的梯度相关只有 0.724；把 softplus β = 1 的网络当分类器用，准确率只有 0.536 —— 它解释的是另一个模型。 Same conclusion as §5.2: the most reliable route to stable explanations is to make the model smooth or robust, not to smooth the explanation afterwards. The CMU group later made this precise: Wang, Wang, Ramkumar, Fredrikson, Mardziel & Datta (NeurIPS 2020, “Smoothed Geometry for Robust Attribution”) tie attribution robustness to the geometry of the decision surface, propose a post-hoc fix (averaging gradients over a uniform neighborhood) and a training-time regularizer that penalizes the input Hessian’s largest eigenvalue.

8.4 Fooling LIME and SHAP (Slack et al. 2020)

The second adversary owns the model. Slack, Hilgard, Jia, Singh & Lakkaraju (AIES 2020, “Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods”) exploit the point Lecture 2 §7.3 raised: perturbation-based explainers query the model on off-manifold points, where nobody checks its behavior. The scaffolding attack has three parts:

1. a biased model $f_{\text{bias}}$ — what is really applied to people;
2. an innocuous model $f_{\text{innoc}}$ — what the explainer should see;
3. an out-of-distribution detector trained to tell real inputs from the explainer’s perturbations.

The deployed model is $e(x) = f_{\text{bias}}(x)$ if the detector says “real”, else $f_{\text{innoc}}(x)$.

代码 05 演示 3 在第 1 讲的信贷世界里搭了一个：模型看得到群体标签；$f_{\text{bias}}$ = “group 0 批准、group 1 拒绝”；$f_{\text{innoc}}$ = 按第 1 讲 Model B 的配方（只用 5 个正当特征的逻辑回归）在这份数据上训练的模型；检测器是第 2 讲的梯度提升树，训练数据是真实申请人 vs LIME 式扰动和 KernelSHAP 式混合输入。解释器就是第 2 讲我们自己写的 lime_tabular 和 kernel_shap_sampling，原样导入。

检测器在没见过的数据上判为”真实”的比例：真实申请人 0.991，LIME 式扰动 0.013，SHAP 式混合输入 0.953。真实部署申请人上的行为：

前 40 个部署申请人的解释（group 排第一的比例；平均名次，1 = 最重要，6 = 最不重要）：

• **LIME 被彻底骗过。**对脚手架模型，group 一次都没排第一 —— 平均名次 6.00，最后一名 —— 第一特征是 income，和无害的 Model B 一模一样。LIME 的高斯扰动把 group 变成 0.37、1.2 这样的值，把逾期次数变成小数；检测器认出了其中 98.7%。
• KernelSHAP 在 τ = 0.5 时没有被骗过（在这个世界里）。SHAP 混合输入的每个字段都是某个真实申请人的真实值，第 1 讲世界里特征之间的相关又很弱，检测器把 95.3% 的混合输入当成了真人 —— 它们得到有偏规则，偏见于是显露出来。Slack et al. 也发现 SHAP 比 LIME 难骗。

但攻击者可以拿歧视的强度换隐蔽性：提高阈值 τ，只让”非常像真实”的输入得到有偏规则（τ 取成检测器在留出的真实申请人上的分数的 q 分位数）：

q = 0.5 时，只有一半的真实申请人得到有偏规则，批准率差距却仍有 0.663 —— Model B 的三倍；而 KernelSHAP 只在 7.5% 的申请人上把 group 排第一（平均名次 3.10）。SHAP 并没有丢掉偏见，但它不再是最显眼的特征；只读第一条理由的审计会漏掉它。

8.5 Defending explanations

The deepest lesson is the one from Lecture 2 §9.4: an explanation is evidence, not a certificate. Certified robustness for explanations is an active research area (e.g., bounding how much an attribution can change in a ball, the attributional analogue of §6), but for now the robust practice is to treat explanations as one input to an audit, alongside behavioral metrics that the model owner cannot game.

9. 全景：什么样的声明有根据 · The big picture

两条贯穿全讲的线：

1. Upper bounds vs lower bounds. Attacks bound robustness from above; certificates from below. A defense is only as credible as the gap between them is small. Lecture 1’s linear model closes the gap exactly; deep networks need either expensive search, special training (GloRo), or randomization (smoothing) to close it.
2. Certify what you can, abstain on the rest. Lecture 1 §7.3 → GloRo’s ⊥ class → smoothing’s abstention. In every case robustness is bought with coverage, and the abstention rate is a number that must be reported.

10. 小结 · Summary

六个要点：

1. 防御是关于最坏情况的声明。它需要一个写清楚的 threat model 和最强的自适应攻击；任何一次攻击都只是从上方给鲁棒准确率一个界。
2. **梯度失灵不是鲁棒。**量化、distillation、随机化都骗过了作者自己的攻击，BPDA、logit 损失、EOT 把它们揭穿。黑盒攻击比白盒更强时，坏掉的是评估。
3. 对抗训练有效，也有代价。它必须攻击当前的模型（静态增强会失败），训练代价成倍增加，并且用干净准确率换鲁棒性 —— 有时这种交换在原理上躲不开（Tsipras）。
4. **鲁棒性和可解释性相互关联。**鲁棒模型的梯度指向真正的信号；脆弱的解释来自脆弱的、曲率很大的模型。
5. **证书给出下界，但要付代价。**精确方法不扩展；便宜的界很松，除非模型为它而训练（GloRo）；smoothing 能证明任何网络，但只是很小的 $L_2$ 半径、概率性的，还要弃权。
6. **解释也能被攻击。**用自适应攻击评估解释的防御，优先使用 on-manifold 的解释，并且永远直接审计行为 —— 脚手架模型能躲过 LIME，躲不过批准率差距。

11. 常见误区 · Common pitfalls

• “我们的防御在 PGD 下有 90%，所以是鲁棒的。” 先查梯度掩蔽：梯度恰好为 0 的比例、迁移攻击是否比白盒更强、ε 很大时攻击是否仍然失败（§3.2）；然后跑针对机制的自适应攻击。
• “我们在训练时 / 测试时加了噪声，所以模型鲁棒。” 随机扰动是平均情况（§1.3、§4.1）。测试时的随机性会被 EOT 攻破（§2.3）；只有配上一个定理，它才成为防御（§7）。
• “我们用对抗样本训练过了。” 用的是哪些？针对旧模型一次性生成的样本，只会教模型认出这批样本（对它们 0.998，换一次新攻击只剩 0.012，§4.1）。
• “证书没证出来，所以这个点不鲁棒。” 可靠但不完备的证书（IBP、Lipschitz、FGP 的 unknown、smoothing 的弃权）失败什么也说明不了；只有一个真的对抗样本才能证明不鲁棒。
• “认证准确率比 PGD 下的准确率低，所以证书错了。” 本来就应该这样：认证 ≤ 真实 ≤ 攻击后。真正的错误是反过来 —— 一个被认证的点被攻破了（§6.4、§7.4 都检查了这一点，结果是 0）。
• “Randomized smoothing 证明了我的网络。” 它证明的是平滑分类器 $g$。部署的必须是 $g$（连同蒙特卡洛的代价和弃权），而且半径是 $L_2$ 半径、以 1 − α 的概率成立。
• “只要技巧够多，鲁棒性和准确率总能同时最大化。” 当有预测力的特征恰好不鲁棒时不行（§5.1）：要鲁棒，就得放弃它们。
• “SmoothGrad / β-smoothing 解决了解释的脆弱性。” 要用直接攻击平滑后解释的自适应攻击来检验，还要检查平滑后的解释是否仍然描述你的模型（§8.3）。
• “LIME 和 SHAP 都说模型没用群体信息。” 它们报告的是模型在它们的查询上的行为。要在真实数据上直接审计结果（§8.4）。

12. 练习 · Exercises

练习 1（线性模型的对抗训练 = 对偶范数惩罚）. For $z(x) = w^\top x + b$ and the threat model $B_p(x, \epsilon)$: (a) Show $\min_{\delta \in B_p(0, \epsilon)} s\,(w^\top(x + \delta) + b) = s\,z(x) - \epsilon\lVert w\rVert_*$. (b) Write the adversarial training objective for logistic loss. (c) Explain, using (b), why $L_\infty$ adversarial training in §5.1 moved weight from the 100 weak features to $x_1$. (d) What would change under an $L_2$ threat model?

(a) 由 Hölder 不等式 $|w^\top\delta| \le \lVert w\rVert_*\lVert\delta\rVert_p \le \epsilon\lVert w\rVert_*$，且等号可以取到（$L_\infty$ 时 $\delta = -\epsilon s\,\mathrm{sign}(w)$；$L_2$ 时 $\delta = -\epsilon s\,w/\lVert w\rVert_2$）。(b) $\min_{w,b} \frac1n\sum_i \log\big(1 + \exp(-s_i z(x_i) + \epsilon\lVert w\rVert_*)\big)$ —— 这就是 train_linear_adv 的目标。(c) $L_\infty$ 的对偶范数是 $\lVert w\rVert_1$：100 个各 0.05 的权重付出 $\epsilon\cdot 5$ 的代价，而一个大小为 2 的 $w_1$ 只付出 $2\epsilon$，同时 $x_1$ 每个样本贡献的 margin 是 $|w_1|$。ε 足够大时，用弱特征换来的 margin 抵不上 $\epsilon\lVert w_{\text{weak}}\rVert_1$ 的惩罚。(d) 惩罚变成 $\epsilon\lVert w\rVert_2$：它不偏好”稀疏”，100 个弱特征的 $L_2$ 范数只有 $0.05\cdot\sqrt{100} = 0.5$，所以 $L_2$ 对抗训练不会这么彻底地放弃弱特征 —— 同一个分布在不同 threat model 下的鲁棒 / 准确权衡不同。这也对应第 1 讲练习 3：$\lVert w\rVert_1/\lVert w\rVert_2$ 随维数增长。

练习 2（Danskin 与”错误的最大值点”）. (a) In one or two sentences, why does Danskin’s theorem justify “attack, then take a gradient step on the attacked input”? (b) Static augmentation (§4.1) also takes gradient steps on adversarial inputs. Which assumption of Danskin’s theorem does it violate? (c) FGSM-based adversarial training sometimes shows catastrophic overfitting: FGSM accuracy on training data goes up while PGD accuracy collapses to near 0. Explain this with the same lens.

(a) 外层目标是 $\phi(\theta) = \max_\delta \ell(\theta, \delta)$；Danskin 说（在最大值点唯一等正则条件下）$\nabla\phi(\theta) = \nabla_\theta\ell(\theta, \delta^\star)$。所以只要内层真的找到了 $\delta^\star$，在 $x + \delta^\star$ 上的普通梯度就是鲁棒损失的梯度。(b) 它用的 $\delta$ 是旧参数 $\theta_0$ 的最大值点，不是当前 $\theta$ 的。训练几步之后 $\theta$ 变了，这些 $\delta$ 已经不是最坏情况 —— 模型学会了在这些固定点上降低损失（0.998），而真正的 $\max_\delta$ 并没有下降（0.012）。(c) FGSM 是对 $\delta^\star$ 的一阶近似，只在损失面近似线性时准确。模型可以学会让损失面在 $x$ 附近弯曲（梯度方向一步走过去损失不高，但多走几步损失很高），这样 FGSM 找到的”最大值点”越来越差，Danskin 的前提失效，梯度方向不再指向鲁棒性。Wong et al.（2020）发现随机初始化 + 合适的步长可以避免这一点；PGD 多步内层优化本身就更不容易被”骗”。

练习 3（randomized smoothing 的算术）. σ = 0.25, α = 0.001. (a) With $n = 1000$ and $k = 990$ votes for class A, the code gives $\underline{p_A} = 0.9760$. What radius is certified? (b) How large must $n$ be (with all votes for A) to certify $R = 0.75$? (c) Why does a larger $n$ never help an image whose true $p_A$ is 0.7 reach $R = 0.25$?

(a) $R = 0.25\cdot\Phi^{-1}(0.9760) = 0.25\times1.978 \approx 0.495$。(b) 需要 $\Phi^{-1}(\underline{p_A}) \ge 3$，即 $\underline{p_A} \ge \Phi(3) = 0.99865$；全票时 $\underline{p_A} = \alpha^{1/n}$，所以 $n \ge \ln\alpha/\ln 0.99865 \approx 5114$。(c) $n \to \infty$ 时 $\underline{p_A} \to p_A = 0.7$，半径收敛到 $0.25\cdot\Phi^{-1}(0.7) = 0.25\times0.524 = 0.131 < 0.25$。样本数只能减少估计误差，不能改变 $f$ 在噪声下的行为；要更大的半径，只能让基分类器在噪声下更”坚决”，或者换更大的 σ（代价是准确率）。

练习 4（GloRo 的多分类版本）. For $C$ classes with logits $z_c(x) = w_c^\top h(x) + b_c$, where $h$ is the penultimate layer with Lipschitz bound $K_h$. (a) Show $K_{ij} = \lVert w_i - w_j\rVert_2 \cdot K_h$ is a valid Lipschitz bound for $z_i - z_j$. (b) Why is this tighter than using $2\max_c\lVert w_c\rVert_2\cdot K_h$? (c) Show that if ⊥ does not win, the prediction is certified at radius ε. (d) In the binary case with logits $(0, z)$, recover $|z| > \epsilon K$.

(a) $z_i - z_j = (w_i - w_j)^\top h(x) + \text{const}$；线性映射的 Lipschitz 常数是 $\lVert w_i - w_j\rVert_2$，与 $h$ 的 $K_h$ 相乘。(b) 由三角不等式 $\lVert w_i - w_j\rVert \le \lVert w_i\rVert + \lVert w_j\rVert \le 2\max\lVert w_c\rVert$；两个相似的类（权重接近）的差的范数可以远小于它们各自的范数。(c) ⊥ 不赢 ⇒ 对所有 $i \ne j$，$z_j(x) > z_i(x) + \epsilon K_{ij}$；在球内任一点 $x'$，$z_j(x') - z_i(x') \ge z_j(x) - z_i(x) - K_{ij}\lVert x' - x\rVert > \epsilon K_{ij} - \epsilon K_{ij} = 0$，所以 $j$ 仍然是最大的。(d) $j = 1$（$z > 0$）时 ⊥ $= 0 + \epsilon K$，⊥ 不赢 ⇔ $z > \epsilon K$；$j = 0$ 时 ⊥ $= z + \epsilon K$，不赢 ⇔ $-z > \epsilon K$。合起来 $|z| > \epsilon K$。

练习 5（读一份防御报告）. A paper reports, on CIFAR-10 at ε = 8/255: clean 91%; PGD-20 white-box 78%; transfer from a standard ResNet 52%; PGD at ε = 64/255: 41%; gradient-free (query-based) attack 49%. The defense passes inputs through a learned denoiser before classification. (a) Which red flags do you see? (b) Which category of obfuscation is most likely, and what adaptive attack would you try first? (c) What would you need to see before believing a robust accuracy above 50%?

(a) 三面红旗：迁移（52%）和无梯度攻击（49%）都比白盒（78%）强；ε = 64/255 是一个能让图像面目全非的预算，攻击后仍有 41%，攻击没有接近 100% 成功；（隐含地）白盒攻击弱于”不看梯度”的攻击，说明梯度没用。(b) 学习型去噪器通常是很深的网络，可能导致 vanishing / exploding 或 shattered（如果有量化、截断）梯度。先试 BPDA（把去噪器的反向近似为恒等函数，或对端到端网络直接反传但换 logit 损失），以及对去噪器 + 分类器整体做攻击；如果去噪器有随机性，加 EOT。(c) 至少：对整个系统的自适应攻击（含 BPDA / EOT）、步数与重启的收敛曲线、ε–成功率曲线趋向 100%、AutoAttack 结果、公开代码与模型；理想情况下还有在 RobustBench 上的独立复现。即便如此，它仍是上界；想要下界只能靠证书。

练习 6（脚手架攻击的攻与防）. (a) In §8.4, why did KernelSHAP resist at τ = 0.5 while LIME didn’t? Answer in terms of what each method queries. (b) Suppose the world had a hard constraint, e.g. age ≥ 19 + credit-history length, and age were a model feature. Would SHAP become easier or harder to fool? Why? (c) Design an audit, using only black-box queries and real applicants, that detects a scaffold without looking at any explanation. (d) Why is “compute group approval rates on real data” not a complete defense either?

(a) LIME 查询 $x$ 加高斯噪声后的点：群体变成连续值、计数变成小数、特征间关系被打乱，检测器一眼就认出来（98.7%）。KernelSHAP（interventional）查询”一部分字段来自 $x$、其余来自某个真实背景样本”的混合输入：每个字段都是真实值，在特征之间相关很弱的世界里，混合输入几乎就是一个真实的人（95.3% 被判为真实），于是它们得到有偏规则。(b) 更容易骗。约束越强，混合输入越容易违反约束（比如 history 来自 $x$、age 来自背景样本，导致 age < 19 + history），检测器就越能把它们认出来。讽刺的是：数据越”有结构”，interventional 解释越 off-manifold，也越容易被利用（第 2 讲 §7.3）。（编写本讲时的一个原型加入了这样的 age 特征：SHAP 把 group 排第一的比例从 1.00 降到 0.88 —— 方向如此，但效果不大。）(c) 例如：对每个真实申请人，查询它的若干个”语义不变”的变体（把计数加 0.01、把收入乘 1.001 这类不改变实质的扰动），比较模型在真实点与这些点上的决定；脚手架会在两者之间系统性地跳变，而诚实模型几乎不变。或者：对真实申请人和 LIME 扰动样本分别计算按群体的批准率，两个分布上的行为差异本身就是证据。(d) 它能发现”结果有差距”，但不能区分差距来自世界的差异、合理的特征，还是刻意的歧视（第 1 讲 §5.2、第 5 讲）；它需要群体标签和真实的部署数据；而且一个更聪明的对手可以在审计用的数据上表现正常（如果审计数据可以被识别出来）—— 这又回到了”审计的查询分布能否被识别”的问题。

代码相关的练习（“动手改一改”）见 code/lecture08/README.md。

13. 延伸阅读 · Further reading

本周的阅读材料（Spring 2022 Week 7）

• Madry, Makelov, Schmidt, Tsipras & Vladu, “Towards Deep Learning Models Resistant to Adversarial Attacks”, ICLR 2018.
• Fromherz, Leino, Fredrikson, Parno & Păsăreanu, “Fast Geometric Projections for Local Robustness Certification”, ICLR 2021.
• Dombrowski, Alber, Anders, Ackermann, Müller & Kessel, “Explanations can be manipulated and geometry is to blame”, NeurIPS 2019.
• Ross & Doshi-Velez, “Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients”, AAAI 2018.
• Slack, Hilgard, Jia, Singh & Lakkaraju, “Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods”, AIES 2020.

评估与梯度掩蔽

• Papernot, McDaniel, Wu, Jha & Swami, “Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks”, IEEE S&P 2016；Carlini & Wagner, “Defensive Distillation is Not Robust to Adversarial Examples”, 2016；“Towards Evaluating the Robustness of Neural Networks”, IEEE S&P 2017.
• Athalye, Carlini & Wagner, “Obfuscated Gradients Give a False Sense of Security”, ICML 2018；Athalye, Engstrom, Ilyas & Kwok, “Synthesizing Robust Adversarial Examples”（EOT）, ICML 2018.
• Carlini et al., “On Evaluating Adversarial Robustness”, arXiv 2019；Tramèr, Carlini, Brendel & Madry, “On Adaptive Attacks to Adversarial Example Defenses”, NeurIPS 2020.
• Croce & Hein, “Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks”（AutoAttack）, ICML 2020；Croce et al., RobustBench.

对抗训练与鲁棒性 vs 准确率

• Zhang et al., “Theoretically Principled Trade-off between Robustness and Accuracy”（TRADES）, ICML 2019；Rice, Wong & Kolter, “Overfitting in Adversarially Robust Deep Learning”, ICML 2020；Wong, Rice & Kolter, “Fast is Better than Free”, ICLR 2020.
• Tsipras, Santurkar, Engstrom, Turner & Madry, “Robustness May Be at Odds with Accuracy”, ICLR 2019；Schmidt et al., “Adversarially Robust Generalization Requires More Data”, NeurIPS 2018；Santurkar et al., “Image Synthesis with a Single (Robust) Classifier”, NeurIPS 2019.

认证

• Katz et al., “Reluplex”, CAV 2017；Tjeng, Xiao & Tedrake, “Evaluating Robustness of Neural Networks with Mixed Integer Programming”, ICLR 2019.
• Wong & Kolter, “Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope”, ICML 2018；Gowal et al., “On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models”, 2018；Zhang et al., “Efficient Neural Network Robustness Certification with General Activation Functions”（CROWN）, NeurIPS 2018.
• Tsuzuku, Sato & Sugiyama, “Lipschitz-Margin Training”, NeurIPS 2018；Leino, Wang & Fredrikson, “Globally-Robust Neural Networks”, ICML 2021.
• Cohen, Rosenfeld & Kolter, “Certified Adversarial Robustness via Randomized Smoothing”, ICML 2019；Lecuyer et al., “Certified Robustness to Adversarial Examples with Differential Privacy”, IEEE S&P 2019；Salman et al., “Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers”, NeurIPS 2019.

解释的鲁棒性

• Ghorbani, Abid & Zou, “Interpretation of Neural Networks is Fragile”, AAAI 2019.
• Wang, Wang, Ramkumar, Fredrikson, Mardziel & Datta, “Smoothed Geometry for Robust Attribution”, NeurIPS 2020.

讲座：CMU 18-739 —— Dimitris Tsipras, “Robust Machine Learning: Progress, Challenges, Humans”；Nicholas Carlini, “Towards Evaluating the Robustness of Neural Networks”。

14. 下一讲预告 · Next lecture

Lecture 9：隐私攻击（Privacy Attacks on ML）。

这一讲的对手想让模型说错；下一讲的对手想让模型说出不该说的。一个训练好的模型是训练数据的函数，它会携带训练数据的信息：第 1 讲 §5.4 用一个 loss 阈值就做出了成员推断（membership inference）。第 9 讲从 linkage 与去匿名化（Sweeney、Netflix）讲起，到 model inversion、shadow models（Shokri et al. 2017）、LiRA 与”低 FPR 下的 TPR”、memorization 与 LLM 的训练数据提取。和本讲一样，评估攻击时最重要的问题仍然是”攻击够不够强”：一个失败的攻击不能证明没有泄露 —— 正如本讲里一个失败的 PGD 不能证明鲁棒。对应于”证书”的东西是第 10 讲的差分隐私：一个对任何攻击都成立的上界。