Lecture 7 · 对抗样本与攻击 · Adversarial Examples & Attacks

对应 Maps to：Stanford CS329T Spring 2022 Week 6 —— robustness 模块第一周（Anupam Datta & John Mitchell 的 “Robustness” 讲座：adversarial attacks、real-world adversarial attacks、“戴一副古怪的眼镜就能让模型说出一个明星的名字”），fireside chat 讨论 Ilyas et al. 2019 “Adversarial Examples Are Not Bugs, They Are Features”；阅读材料 Papernot et al. 2016 “The Limitations of Deep Learning in Adversarial Settings”（JSMA）、Carlini & Wagner 2017 “Towards Evaluating the Robustness of Neural Networks”、Madry et al. 2018 “Towards Deep Learning Models Resistant to Adversarial Attacks”、Sharif et al. 2016 “Accessorize to a Crime”，以及 CMU 18-739 中 Nicholas Carlini 讲 C&W 攻击的 slides；HW5 Part 1（PGD；Carlini & Wagner L2 攻击 —— 我们拿到的是 Spring 2021 版的作业说明，2022 年沿用）、Lab 6（robustness、PGD、CleverHans）。Fall 2023 Lecture 6 “Project Design and Planning” 中的 robustness 部分（Ribeiro et al. 2018 “Semantically Equivalent Adversarial Rules”、Ribeiro et al. 2020 “Beyond Accuracy: Behavioral Testing of NLP Models with CheckList”）。 配套代码 Code：code/lecture07/（PyTorch，CPU；5 个脚本 + 2 个工具模块，每个脚本 26–50 秒） 预计阅读时间：3 小时（含运行代码）

0. 本讲导航 · How to read this lecture

0.1 Learning goals

After this lecture you should be able to:

1. 写出一个 threat model 的三个组成部分 —— 攻击目标（untargeted / targeted）、攻击者的知识（white-box / transfer / score-based / decision-based）、攻击者的能力（$L_\infty$ / $L_2$ / $L_0$ 球、贴纸、物理变换、语义改写），并说明为什么”鲁棒”这个词离开 threat model 就没有意义。
2. Derive FGSM from a first-order Taylor expansion and the dual norm, and explain — building on Lecture 1 §5.3 — why high-dimensional inputs make worst-case perturbations so much more effective than random ones.
3. Implement and tune PGD (projection, step size, random start, restarts), explain why Madry et al. call it a “universal” first-order adversary, and use it to report robust accuracy curves honestly.
4. 解释 Carlini & Wagner 攻击的每一个设计选择（罚项而不是约束、tanh 换元、logit 间隔的 $f$、截断在 $-\kappa$、对 $c$ 二分搜索），以及它为什么攻破了 defensive distillation。
5. Explain transferability and black-box attacks (substitute models, NES / SPSA, decision-based), and why keeping a model secret is not a defense.
6. 解释 物理世界攻击 与 Expectation over Transformation（EOT）；理解 Ilyas et al. 的 “features, not bugs” 论点、它的实验（$D_{\text{rand}}$、$D_{\text{det}}$）和争议；知道 NLP 里的鲁棒性为什么要用 CheckList 式的行为测试和语义等价改写来评估。

0.2 先修与本讲范围 Prerequisites & scope

0.3 本讲地图 Map

§1  一个对抗样本：同样大小的随机噪声没事，顺着梯度就出事                ← 代码 01 演示 1
§2  Threat model：目标 × 知识 × 能力；"小"用哪把尺子量？               ← 代码 02 演示 6
§3  为什么会存在：线性解释、高维、其他假说                            ← 代码 01 演示 2–3
§4  FGSM：一步线性化                                                 ← 代码 01 演示 2–3
§5  BIM / PGD：多步 + 投影；步长、重启、定向、L∞ vs L2                  ← 代码 01 演示 3–6
§6  Carlini & Wagner：c、tanh、f、κ；防御性蒸馏为什么失败               ← 代码 02 演示 1–5
§7  JSMA（L0）与 DeepFool（最小 L2）                                   ← 代码 02 演示 2、6
§8  怎样评估攻击与鲁棒性
§9  迁移攻击与黑盒攻击（替身、NES、decision-based）                     ← 代码 03
§10 物理世界：打印拍照、眼镜框、停车标志、EOT、贴纸                       ← 代码 04 演示 1–2
§11 "Not bugs, features"：非鲁棒特征与 D_rand / D_det                  ← 代码 05
§12 NLP：离散输入、SEARs、CheckList、同义词替换                        ← 代码 04 演示 3–4
§13 桥梁：分布漂移（第 1 讲）、LLM 越狱（第 15 讲）、防御（第 8 讲）
§14–18 小结、误区、练习、延伸阅读、下一讲

0.4 Notation

0.5 关于数字 · About the numbers

讲义里所有实验数字都来自配套代码的实际输出（固定随机种子，PyTorch 单线程，同一台机器上重复运行逐字相同）。数据全部由程序生成：16 × 16 的七段数码管数字（adv_data.py），每张图有随机的大小、粗细、位置、斜体和亮度，再加噪声 N(0, 0.08²)。被攻击的 Model A 是一个约 1 万参数的小 CNN，干净测试准确率 0.998。为什么用七段数码管？因为类别之间的”语义距离”是已知的 —— 7 和 1 只差顶上一横，8 和 0 只差中间一横 —— 这让我们能把对抗扰动和真正改变了数字的编辑放在一起比较。数字只说明机制，不代表任何真实系统。和计划不一致的结果都记录在代码 README 的”关于实验结果”里。

1. 一个对抗样本 · An adversarial example

1.1 The phenomenon

Szegedy et al. (2014), “Intriguing properties of neural networks”, reported that image classifiers with excellent test accuracy can be made to misclassify by perturbations so small that a human sees no difference — and that the same perturbed image often fools other networks too. Goodfellow, Shlens & Szegedy (2015) made it famous with the panda that became a “gibbon” with 99.3% confidence (Lecture 1 §3.3), and gave a one-line method to find such perturbations: FGSM (§4).

我们的版本（代码 01 演示 1）：测试图 #0 是一个 “1”。分别加上三种扰动，每个像素的改动都不超过 0.08（像素范围是 0 到 1）：

三种扰动一样”大”（$L_\infty$ 都是 0.08，$L_2$ 都约 1.0），唯一的区别是方向：随机噪声让模型稍微不那么确定，顺着 loss 梯度的扰动让它以 98% 的置信度说 “4”。字符画（代码输出里）显示扰动几乎铺满整张图，而对抗样本在人眼看来仍是同一个 “1”。

1.2 为什么要关心 · Two reasons to care

Carlini 的讲座（CMU 18-739 用的 slides）把理由分成两类：make ML robust 和 make ML better。

1. Security（安全）：只要输入可能来自一个有意图的人 —— 垃圾邮件、恶意软件、人脸识别门禁、自动驾驶看到的路牌、内容审核、作业自动评分（Fall 2023 第 6 讲的例子：“学生能不能稍微改一下错误答案，让 LLM 判它对？”）—— 平均情况下的准确率就不再是相关的指标。对手会专挑最坏的情况。
2. Science（理解）：一个人类完全看不出区别、模型却完全改变主意的输入，说明模型用来做决定的东西和我们以为的不一样。§11 会看到，这些”东西”可能是数据里真实存在、对预测有用、但人类感知不到的特征。这和第 2–4 讲的问题是同一个问题：“模型到底依赖什么？”

Spring 2022 robustness 模块的开场 slide 列出了这个模块的主线，正好是本讲和第 8 讲的分工：

• ML models can be very brittle: a small change in the input can produce a very different output.
• Adversarial examples can be produced systematically（本讲）。
• There are systematic methods to improve robustness; adversarial learning takes small changes into account（第 8 讲）。
• There are limitations: more robust models may be less accurate; and intuitive measures of “small change in input” are tricky — there is no clear metric for human perception（§2.3）。

2. Threat model · 威胁模型

2.1 The formal object

Robustness is always “robustness to something”. The standard formalization replaces the expected loss with an expected worst-case loss over a perturbation set $\Delta(x)$:

This is the robustness row of Lecture 1 §2.2. Two equivalent ways to report a model’s robustness at a point $x$:

• fixed budget: is there a $\delta$ with $\lVert\delta\rVert \le \epsilon$ that flips the prediction? → robust accuracy at $\epsilon$;
• minimal perturbation: how large must $\delta$ be? → $\rho(x) = \min\{\lVert\delta\rVert : \hat y(x + \delta) \ne y\}$, and robust accuracy at $\epsilon$ is just $P(\rho(x) > \epsilon)$.

Neither can be computed exactly for a deep network in general (the inner max is non-concave; exact verification is NP-hard for ReLU networks — Katz et al. 2017). An attack finds some $\delta$, so it gives an upper bound on robust accuracy (or on $\rho$). A certificate (Lecture 8) gives a lower bound.

2.2 目标 × 知识 × 能力 · Goal × knowledge × capability

两个说明。第一，threat model 是一个承诺：一个”在 $L_\infty$, $\epsilon = 8/255$ 下鲁棒”的模型对 $L_2$ 攻击、贴纸、旋转可能毫无抵抗力（§5.5 和第 8 讲）。第二，知识越少不代表越安全 —— §9 会看到，迁移攻击只需要一个自己训练的替身模型。

2.3 “小”用哪把尺子量？ · Which “small”?

Spring 2022 的讲座留了一组讨论题：“Do metrics match human perceptibility? $L_0$ — how many pixels changed at all? $L_1$? $L_2$? $L_\infty$ — max change of any single pixel?” 以及 “What if ‘different’ includes the semantics of what is pictured?”。七段数码管让我们可以测量这个问题：用同一个字体、同一份噪声，少画一段 —— 真实类别就变了（7 → 1、8 → 0……）。代码 02 演示 6 比较这种”语义编辑”和同一批图上的最小对抗扰动（各自在自己的尺子下最小）：

• $L_\infty$ 把”对抗”和”语义”分得最开：真正改变数字的编辑要把某些像素改 0.66 以上，对抗扰动每个像素只改 0.043。这就是为什么图像上的对抗样本研究最常用 $L_\infty$。
• $L_0$ 几乎分不开：贪心 $L_0$ 攻击中位数只改 2 个像素，而它找到的例子把一个 8 中间一横的两个像素擦掉了（§7.1）—— 在笔画只有 1–2 像素粗的图上，“改两个像素”本身就可能改变人眼看到的东西。
• 没有一把尺子对所有数据都对。在 MNIST 上 $\epsilon = 0.3$ 的 $L_\infty$ 是常用的设定，在自然图像上通常是 $8/255 \approx 0.03$；对文本，“小”根本不是一个范数（§12）。先写下 threat model，再谈鲁棒性。

2.4 Two ways to evaluate robustness

Carlini’s slides put it bluntly: you can (1) construct a proof of robustness — possible for specific points, small networks, simple datasets (Lecture 8); or (2) demonstrate a constructive attack. This lecture is about (2). An attack that fails proves nothing about robustness; an attack that succeeds proves non-robustness. That asymmetry is why evaluation (§8) matters so much.

3. 为什么会存在对抗样本 · Why do adversarial examples exist?

3.1 The linear explanation（第 1 讲 §5.3 的延续）

Goodfellow et al. (2015) argued that adversarial examples are not caused by “too much nonlinearity” but by too much linearity in high dimensions. Recall Lecture 1, exercise 3: for a linear score $w^\top x$ with $d$ weights, the perturbation $\delta = \epsilon\,\text{sign}(w)$ changes the score by $\epsilon\lVert w\rVert_1$, while a random-sign perturbation of the same size changes it by about $\epsilon\lVert w\rVert_2$. For Gaussian weights the ratio is $\sqrt{2d/\pi}$ — 22 for MNIST-sized inputs, 309 for ImageNet. Many tiny changes, all pushing the same way, add up.

A deep network is not linear, but locally it is close to linear (ReLUs are piecewise linear, and networks are trained to be smooth enough to optimize). So the same argument applies to its input gradient $g$. 代码 01 演示 2 在 Model A 上测量：

• $\lVert g\rVert_1 / \lVert g\rVert_2$ 的中位数是 9.61。上限是 $\sqrt d = 16$（所有 $|g_j|$ 相等），高斯随机向量约为 $\sqrt{2d/\pi} = 12.77$。梯度比高斯向量更集中（集中在笔画附近），但仍然远大于 1：最坏方向比典型方向强约 10 倍。
• 一阶近似 $\ell(x + \delta) \approx \ell(x) + g^\top\delta$ 能用多远？

ε 很小时线性化很准；ε 变大，实际的 loss 增量远超线性预测（ε = 0.1 时 23 倍）。原因：干净样本的 loss 很小（0.0146），模型很有把握，softmax 处在饱和区，梯度被压扁了；一旦扰动把 logit 间隔推近 0，cross-entropy 开始随间隔线性增长。所以线性化是找方向的好工具，但它低估了沿这个方向走下去的危害。随机方向的增量在 ε ≤ 0.03 时几乎为 0 —— 正负相消。

3.2 最坏方向比随机方向省多少预算？

同一件事的另一种说法：每个样本”翻转所需的最小 ε”（代码 01 演示 3，二分搜索，只统计干净时判对的样本）：

随机方向需要约 5 倍的预算。比第 1 讲线性模型的 $\sqrt{2d/\pi} \approx 12.8$ 倍小，有两个原因：网络不是线性的（远离 $x$ 时梯度方向会变）；而且背景像素已经是 0，随机的 “−ε” 被裁剪到 [0, 1] 里时直接消失了 —— 随机扰动在黑背景上大约只有”+ε”的那一半在起作用。

3.3 Other hypotheses（谨慎地说）

The linear story explains why small $L_\infty$ perturbations can have large effects, but it is not the whole story, and the field has several complementary views:

• High-dimensional geometry. In high dimensions, most of the volume of a set lies near its boundary. Gilmer et al. (2018, “Adversarial Spheres”) built a synthetic problem — two concentric spheres — where any classifier with a small but nonzero error rate must have most of its correctly classified points very close to an error. Shafahi et al. (2019, “Are adversarial examples inevitable?”) gave related bounds. These results show adversarial examples can be unavoidable for some distributions; whether real image distributions are like that is debated.
• Boundary tilting. Tanay & Griffin (2016) argued that adversarial examples arise when the decision boundary lies close to the data manifold but tilted relative to it, so a short step off the manifold crosses it.
• On- vs off-manifold. Many $L_p$ adversarial examples leave the data manifold (our perturbations brighten the black background, which no real digit does). Stutz, Hein & Schiele (2019) separated on-manifold adversarial examples (which they relate to ordinary generalization errors) from off-manifold ones.
• Non-robust features (Ilyas et al. 2019, §11): the perturbations are not exploiting a bug of the model but real, predictive patterns in the data that humans don’t perceive. 这个说法改变了问题的性质：如果对抗脆弱性来自数据本身，那么”更好的拟合”不会消除它，必须告诉模型我们想要哪些特征（第 8 讲的 adversarial training）。

这些假说并不互斥。本讲的态度：把它们当作理解实验结果的透镜，而不是已经定论的理论。

4. FGSM：一步线性化 · The Fast Gradient Sign Method

4.1 Derivation

Linearize the loss around $x$: $\ell(x + \delta, y) \approx \ell(x, y) + g^\top\delta$. Maximize the linear term over the ball:

(this is the definition of the dual norm). The maximizers are:

• $p = \infty$ ($q = 1$): $\delta^* = \epsilon\,\text{sign}(g)$ — every coordinate moves to the boundary of its interval, in the direction of the gradient. This is FGSM: $x' = \text{clip}_{[0,1]}\big(x + \epsilon\,\text{sign}(\nabla_x\ell(x, y))\big)$.
• $p = 2$ ($q = 2$): $\delta^* = \epsilon\, g / \lVert g\rVert_2$ — the fast gradient method (FGM).
• $p = 1$ ($q = \infty$): put the whole budget on the single coordinate with the largest $|g_j|$ — a precursor of $L_0$-style attacks.

The sign is not a heuristic: it is exactly the steepest-ascent direction for the $L_\infty$ geometry. A targeted FGSM descends the loss of the target class instead: $x' = x - \epsilon\,\text{sign}(\nabla_x \ell(x, t))$ (Kurakin et al. 2017 used the “least likely” class as $t$).

4.2 结果：鲁棒准确率 vs ε

代码 01 演示 3，前 500 张测试图（干净准确率 0.998），五种扰动下 Model A 的准确率：

• 随机噪声几乎无害：ε = 0.1 时准确率仍有 0.988。如果你只用随机噪声测鲁棒性，会以为这个模型很鲁棒（第 1 讲常见误区）。
• 一步就很强：FGSM 在 ε = 0.05 时把准确率降到 0.648。
• 多步更强：ε = 0.08 时 FGSM 0.312，BIM / PGD 0.07–0.10。单步的线性化在弯曲的 loss 面上走不到最坏点（§3.1 的表：线性化在 ε ≥ 0.01 时就不准了）。
• 随机起点和重启只多找到 1–3 个百分点 —— 在这个没有防御的模型上，从原点出发的 BIM 已经很接近 PGD 能找到的最坏情况。在有防御（尤其是有梯度遮蔽）的模型上，这两者的差别会大得多（第 8 讲）。

5. BIM 与 PGD · Iterating and projecting

5.1 The algorithm

BIM (Basic Iterative Method, Kurakin, Goodfellow & Bengio 2017) applies FGSM repeatedly with a small step and clips back into the $\epsilon$-ball after each step. PGD (Madry et al. 2018) is the same projected gradient ascent, plus a random starting point inside the ball:

where step$(g) = \text{sign}(g)$ for $L_\infty$ and $g/\lVert g\rVert_2$ for $L_2$, and the projection $\Pi$ is per-coordinate clipping to $[x_i - \epsilon, x_i + \epsilon]$ for $L_\infty$, or rescaling $\delta \leftarrow \delta\cdot\min(1, \epsilon/\lVert\delta\rVert_2)$ for $L_2$. Clipping to $[0, 1]$ after projecting never pushes $\delta$ out of the ball, because $x$ itself is in $[0,1]^d$ (HW5 written exercise 3 asks you to compare the clipping, $L_1$ and $L_2$ projection modes).

5.2 步长与迭代次数（代码 01 演示 4a）

ε = 0.05、40 步、从原点出发，平均 CE loss 沿迭代的变化：

• 步长太小，40 步刚刚开始起作用。Madry et al. 的默认值 $\alpha = 2.5\epsilon/\text{steps}$ 的道理是：累计能走 2.5 个 ε，即使走了弯路也能到达球面上任何一点。
• 步长 = ε 等于”迭代的 FGSM 满步”：第一步就跳到 loss 0.964，然后在角点之间来回跳、停在 1.33，最终准确率反而更高（0.542）。L∞ 球的最坏点不一定在角上，大步子没法细调。

5.3 Random restarts and the “universal first-order adversary”

Madry et al. framed robustness as a saddle-point problem, $\min_\theta \mathbb{E}\big[\max_{\delta \in \Delta}\ell(x + \delta, y; \theta)\big]$, and needed the inner maximization to be solved well enough to train against it (Lecture 8). The inner problem is non-concave, so why trust PGD? Their empirical answer: starting PGD from many random points in the ball (10⁵ in their experiments), the local maxima found have remarkably concentrated loss values, with no isolated, much-higher maxima. They argued that PGD is therefore, in a certain sense, a “universal” adversary among first-order methods: a model robust to PGD should be robust to any attack that only uses gradients.

代码 01 演示 4b 做一个小版本：10 次随机重启（每次 PGD-20），ε = 0.05：

• 每个样本 10 次最终 loss 的变异系数（std / mean）中位数 0.083，90% 分位 0.291 —— 确实很集中。
• 单次重启的平均成功率 0.507，10 次里至少成功一次 0.534，只有 7.2% 的样本”有时成功、有时失败”。

两个 caveat：(1) “first-order” 是关键限定 —— 不用梯度的攻击（§9.3 的 NES）或换一种 loss（§6.6）可能找到 PGD 找不到的东西；(2) 当防御让梯度变得没用时（第 8 讲的 obfuscated gradients），PGD 的 loss 会集中在一个错误的低值上。集中性说明的是”PGD 在这个 loss 面上找到的都差不多”，而不是”loss 面上没有更坏的点”。

5.4 非定向 vs 定向（代码 01 演示 5）

非定向攻击会去哪？ ε = 0.05 的非定向 PGD-20 成功率 0.510（前 300 张），最常见的转移是：5→6（16 次）、8→6（11）、2→8（10）、3→9（8）、0→7（7）、6→8（7）、8→0（7）、9→5（7）。对照七段数码管：5→6 是加上左下竖（e 段），8→6 是去掉右上竖（b），3→9 是加左上竖（f），6→8 加 b，8→0 去中横（g），9→5 去 b —— 大多数正是”只差一段”的邻居。非定向攻击走最近的边界，而模型眼里最近的边界和人眼里最相似的字形是一致的。这也提醒我们：对抗扰动并不总是”随机的噪声”，它常常是一个很淡的、指向邻近类别的语义方向（§11 再回到这一点）。

定向攻击难得多。把每张图推向 $t = (y + k) \bmod 10$，平均成功率：

定向攻击要到达一个指定的区域，而不只是离开当前区域；同样的预算下成功率低得多。§11 构造 $D_{\text{det}}$ 时这一点会变得很重要。

5.5 $L_\infty$ vs $L_2$（代码 01 演示 6）

$L_\infty$ 攻击是稠密的：几乎每个像素都被推到 ±ε（像一层雪花）。$L_2$ 攻击把能量集中在梯度最大的少数像素上（单个像素最多改了 0.234）。在差不多的 $L_2$ 长度下，$L_2$ 攻击的成功率高得多 —— 因为它优化的就是这把尺子。用一种范数测出的鲁棒性，不能拿去回答另一种范数的问题。

6. Carlini & Wagner · 把”找最小扰动”写成一个好的优化问题

6.1 Context: defensive distillation

In 2016, Papernot et al. proposed defensive distillation: train a network at a high softmax temperature $T$, use its soft labels to train a second network at the same temperature, and deploy it at $T = 1$. It appeared to work spectacularly — in Carlini & Wagner’s words, it reduced the success rate of then-current attacks from 95% to 0.5%. Carlini & Wagner (2017) introduced three new attacks ($L_0$, $L_2$, $L_\infty$) that succeed on both distilled and undistilled networks with 100% probability. The paper became the template for “evaluating robustness”: a defense is only as good as the strongest attack you tried.

6.2 From a constrained problem to a penalty（CMU slides 的讲法）

The natural formulation is

but "$\hat y(x') = t$" is a highly non-linear constraint, and gradient descent can’t handle it directly. So rewrite it as $\min d(x, x') + c\cdot g(x')$ with some loss $g$ that is small when $x'$ is classified as $t$. Carlini’s slides walk through what goes wrong with a naive $g$ such as $1 - F(x')_t$ (one minus the target probability):

1. The global minimum may not be adversarial — if $c$ is too small, $d$ wins and the optimum stays on the correct side.
2. The gradient may not point toward the minimum — in saturated regions of softmax, $g$ is flat.
3. The global minimum may not be the minimally perturbed adversarial example — if $c$ is huge, $g$ keeps pulling past the boundary.

The fix is to design $g$ so that (i) its global minimum sits at the decision boundary and (ii) its gradient points toward that minimum.

6.3 The C&W $L_2$ objective

HW5 Definition 1–2 (with $Z$ the logits):

for a targeted attack; for an untargeted attack $f(x') = \max\big(Z_y(x') - \max_{i\ne y} Z_i(x'),\ -\kappa\big)$. Each piece has a reason:

HW5 书面题 2 问：”$f$ 相比 cross-entropy 的好处是什么？” 答案就在第二、三行：logit 间隔不饱和，而且有下界。

6.4 The role of $c$（代码 02 演示 1）

固定 $c$、200 步 Adam、前 100 张测试图：

$c$ 太小，攻击失败；$c$ 足够大之后，再加大只会让扰动变大（在有限的步数内，“越过边界之后再缩回来”来不及完成）。所以 C&W 对每个样本二分搜索 $c$：成功就减小 $c$，失败就增大。

6.5 The choice of $f$（代码 02 演示 3）

The C&W paper compared seven candidate objectives and found the logit-margin one (their $f_6$) best. 同样 200 步、前 60 张测试图，只换 $f$（每格：成功率 / 成功样本的 $L_2$ 中位数）：

三种 $f$ 的数值尺度不同，所以同一个 $c$ 不完全公平；但扫了四个数量级的 $c$，基于概率的目标最多也只有 65–67% 成功。原因是 §6.2 的问题 2：Model A 对干净样本非常有把握，softmax 处在饱和区，概率对输入几乎没有梯度。

6.6 防御性蒸馏为什么失败（代码 02 演示 4）

我们不真的做蒸馏，只模拟它在测试时的效果：蒸馏网络在温度 $T$ 下训练、在 $T = 1$ 下部署，相当于 logit 被放大了约 $T$ 倍（Carlini & Wagner 2016 的分析）。把 Model A 的 logit 乘以 100，预测一个都不变：

logit 放大 100 倍后，softmax 输出在 float32 里被舍入成恰好的 one-hot，cross-entropy 对输入的梯度在 99.5% 的样本上恰好为 0，$\text{sign}(0) = 0$，FGSM 和 PGD (CE) 一个像素都不改。“防御”看起来完美。但决策边界一点没动：直接在 logit 上工作的 PGD (margin) 和 C&W 照样 98–100% 成功。

This is the first and cleanest example of gradient masking: a defense that breaks the attack rather than removing the adversarial examples. Lecture 8 generalizes it (Athalye, Carlini & Wagner 2018, “Obfuscated Gradients Give a False Sense of Security”).（顺便一提：写本讲的 EOT 代码时，clamp(0, 1) 在恰好为 0 的像素上传回 0 梯度，让我自己的攻击”失效”了 —— 梯度遮蔽也可能是攻击者代码里的 bug；见代码 README。）

6.7 Confidence $\kappa$ and transfer（代码 02 演示 5）

另一个模型 = 同结构、同数据、只换随机种子。前 100 张测试图：

“刚好越过边界”的最小扰动对另一个模型基本没用；“深入错误类”的扰动几乎总能迁移。C&W（2017）还用这个办法从另一个方向攻破了蒸馏：在未蒸馏的模型上生成高置信度（大 κ）的对抗样本，它们能迁移到蒸馏过的模型上。

6.8 $L_0$ and $L_\infty$ variants

Carlini’s slides sketch how the other two norms reuse the $L_2$ machinery:

• $L_0$ from $L_2$: solve the $L_2$ problem, find the pixel that changed least (weighted by gradient), freeze it at its original value, re-solve, and repeat until the attack fails. The remaining free pixels are the $L_0$ attack.
• $L_\infty$ from $L_2$: $L_\infty$ is not differentiable-friendly (only the largest coordinate gets a gradient), so penalize $\sum_i \max(|\delta_i| - \tau, 0)$ instead and shrink $\tau$ over rounds.

7. JSMA 与 DeepFool · Two other families

7.1 JSMA：$L_0$ 与显著图

Papernot et al. (2016), “The Limitations of Deep Learning in Adversarial Settings” (a Spring 2022 reading), built the Jacobian-based Saliency Map Attack: compute the Jacobian of the outputs w.r.t. the inputs, form a saliency map that scores how much increasing each input feature raises the target class while lowering the others, and greedily saturate the best pair of features; repeat. On MNIST they reported a 97% success rate while modifying on average 4.02% of the input features. It is the ancestor of all $L_0$ attacks, and its “saliency map” is the same object as the attribution maps of Lecture 3 — used here to change the decision rather than explain it.

我们的简化版（代码 02 演示 6）：每一步用 logit 间隔的梯度预测”把哪个像素改成 0 或 1 能让间隔降得最多”，改它，重新算梯度。前 200 张测试图：100% 成功，改动像素数中位数 2（65 张图改 1 个像素就够，91 张改 2 个，最多 8 个）。只占 2 / 256 ≈ 0.8% 的像素（JSMA 在 MNIST 上是 4%；数据和规则都不同 —— JSMA 的主要设定是把像素调亮，我们允许调亮或调暗 —— 只作量级参考）。代码输出里的例子是一个 “8”：贪心攻击擦掉了中间一横的两个像素，模型就说 “0”。这是 §2.3 的结论的具体样子 —— 在细笔画的图上，$L_0$ 很小的扰动可以就是一次（不完整的）语义编辑。

7.2 DeepFool：走到最近的线性边界

Moosavi-Dezfooli, Fawzi & Frossard (2016) asked for the minimal $L_2$ perturbation directly. For a linear multi-class classifier, the distance from $x$ to the boundary between the current class $\hat y$ and class $k$ is $|f_k| / \lVert w_k\rVert_2$ with $f_k = Z_k - Z_{\hat y}$ and $w_k = \nabla Z_k - \nabla Z_{\hat y}$; the closest boundary is reached by stepping along $w_k$. DeepFool linearizes, takes that step, and repeats until the label changes, then overshoots by a small factor (2%). No $\epsilon$, no $c$, no step-size schedule.

7.3 四种”最小 $L_2$“估计（代码 02 演示 2）

前 200 张测试图：

C&W 在 80.3% 的样本上比 DeepFool 小，但只小约 5%（中位数比值 1.049）。在这个没有防御的小模型上，四种方法几乎一致；C&W 留下的真正遗产是 §6.5 的目标函数设计和 §6.6 的评估教训，而不是在简单模型上多挤出的几个百分点。$L_2 = 0.38$ 有多小？相当于把 16 个像素各调亮约 0.1，或者把一个像素调亮 0.38。

8. 怎样评估攻击与鲁棒性 · Evaluating attacks

8.1 What to report

两个容易出错的地方：(1) 干净时就判错的样本，算”攻击成功”还是不算？报告 robust accuracy 时它们自然算错；报告”攻击成功率”时要说明分母（代码 01 演示 3 的最小 ε 只统计干净时判对的样本）。(2) untargeted 和 targeted 的成功率差别很大（§5.4），不能混用。

8.2 Sanity checks for an attack evaluation

From Carlini et al. (2019), “On Evaluating Adversarial Robustness” (a checklist written after many defenses were broken), the checks that matter most even for a homework-sized experiment:

• Unbounded attacks must reach 100%. If increasing ε never drives accuracy to zero, the attack is broken (e.g., gradients are masked, §6.6).
• Iterative attacks must beat single-step ones, and more iterations / restarts must not hurt. If FGSM beats PGD, suspect the step size or the loss.
• Try a different loss (CE and margin), a gradient-free attack (§9.3), and transfer from a surrogate (§9.1). If a black-box attack beats a white-box one, the white-box gradients are lying.
• Report everything: norm, ε, steps, step size, restarts, number of examples.

Lecture 8 adds AutoAttack (Croce & Hein 2020), a parameter-free ensemble of attacks that became the standard way to report $L_\infty$ / $L_2$ robustness.

9. 迁移攻击与黑盒攻击 · Transferability and black-box attacks

9.1 Transferability

Szegedy et al. (2014) already noticed that adversarial examples crafted on one network often fool another network trained on a different subset of the data. Papernot, McDaniel & Goodfellow (2016) showed transfer even across model families (e.g., from neural networks to logistic regression, SVMs and decision trees). HW5 written exercise 4 asks you to check it with two options: two models trained on disjoint subsets (Option A) or different architectures on the same data (Option B). 代码 03 两种都做，外加”同结构换种子”和 MLP：

在替身上生成对抗样本，不碰 A，看 A 的错误率（前 500 张测试图）：

• 迁移是真实的、而且很强：ε = 0.1 时，三个 CNN 替身上的 PGD 让 A 错 70–85%，而随机噪声只让 A 错 1.6%。换一份独立的训练数据几乎不影响迁移（0.818 vs 0.846）—— 攻击者不需要你的数据，只需要同一个任务的数据。
• 架构越不同，迁移越弱：MLP 替身自己 100% 被骗，却只骗过 A 16–20%。§11 的解释：两个模型用的特征越相似，对抗样本越容易迁移。
• FGSM vs PGD：Kurakin et al. (2017, “Adversarial Machine Learning at Scale”) 在 ImageNet 上观察到单步攻击比迭代攻击迁移得更好（迭代攻击会”过拟合”替身的特殊性）。我们的结果在 CNN 替身上相反（PGD 0.70–0.85 vs FGSM 0.55–0.65），只有在和目标差别最大的 MLP 替身上才是预期的顺序（0.202 vs 0.160）。哪种更好取决于替身和目标有多像；这类经验规律要在你自己的设定里重新检验。
• 集成有帮助（Liu et al. 2017, “Delving into Transferable Adversarial Examples and Black-box Attacks”）：对 4 个替身同时有效的扰动更可能利用它们共有的特征，ε = 0.1 时 0.884，略好于最好的单个替身，尽管其中包含迁移很差的 MLP。

哪些样本容易迁移？（代码 03 演示 3）按 A 自己在干净图上的 logit 间隔分成四组，A2 → A 的 PGD（ε = 0.1）迁移成功率：间隔最小的四分之一 0.988、第二组 0.980、第三组 0.892、间隔最大的四分之一只有 0.552。A 不太有把握的样本，任何方向的扰动都容易把它推过边界。

9.2 Substitute-model attacks

Papernot et al. (2017, “Practical Black-Box Attacks against Machine Learning”) turned transfer into a complete attack on a remote model the attacker can only query for labels: train a local substitute on a small seed set labeled by the target, grow the set with Jacobian-based dataset augmentation (synthesize points where the substitute’s decision changes fastest, query the target for their labels), then craft adversarial examples on the substitute. They attacked classifiers hosted by commercial ML APIs this way. The lesson for practitioners: an API that returns only labels still leaks enough to train a useful substitute.

9.3 Score-based attacks: estimating the gradient from queries

If the attacker can query scores (probabilities), they can estimate the gradient without seeing the model:

• ZOO (Chen et al. 2017): coordinate-wise finite differences, $\partial\ell/\partial x_i \approx [\ell(x + h e_i) - \ell(x - h e_i)]/2h$. A full gradient costs $2d$ queries — 512 here, hundreds of thousands for ImageNet — so ZOO samples coordinates and uses tricks to cut the cost.
• NES (Ilyas, Engstrom, Athalye & Lin 2018): estimate the gradient of a Gaussian-smoothed loss with random directions, $\hat g = \frac{1}{n\sigma}\sum_k [\ell(x + \sigma u_k) - \ell(x - \sigma u_k)]\,u_k$ with antithetic pairs $\pm u_k$, then take a PGD step with $\text{sign}(\hat g)$. The sign only needs the direction to be roughly right.
• SPSA (Uesato et al. 2018, “Adversarial Risk and the Dangers of Evaluating Against Weak Attacks”): the same idea with Rademacher directions; they used it precisely to show that defenses which looked robust to gradient attacks were not.

代码 03 演示 4：NES，每次迭代 50 次查询（25 对 ±u）+ 1 次检查，ε = 0.1，前 200 张测试图：

成功样本的查询次数中位数 766、均值 1093。对照：白盒 PGD-20 0.990（20 次梯度）；在 4 个替身的集成上做迁移攻击、对 A 只查询 1 次就有 0.890。实际攻击常把两者结合：用替身给出初始方向，再用少量查询修正。

9.4 Decision-based attacks

If the API returns only the top-1 label, score-based estimates are impossible. Decision-based attacks start from a point that is already adversarial — for example an image of another class — and walk toward $x$ while staying on the wrong side of the boundary: the Boundary Attack (Brendel, Rauber & Bethge 2018) uses random proposals, HopSkipJump (Chen, Jordan & Wainwright 2020) estimates the boundary’s normal direction from binary decisions. They need more queries (often thousands to tens of thousands), but they show that even a hard-label API is attackable. 代码 README 的练习 5 让你写一个最简单的版本。

9.5 The lesson

“不公开模型参数”不是防御：它把攻击的代价从 20 次梯度变成了几百到几千次查询，或者一个自己训练的替身模型。限制查询速率、检测异常的查询模式可以提高代价，但改变不了”对抗样本存在、而且会迁移”这个事实。

10. 物理世界 · Physical-world attacks

10.1 Printed and photographed

Kurakin, Goodfellow & Bengio (2017, “Adversarial examples in the physical world”) printed adversarial images, photographed them with a cell-phone camera, and fed the photos to an ImageNet Inception classifier: a large fraction remained misclassified. They also observed that the subtler perturbations produced by iterative methods were more likely to be destroyed by the photo transformation than the cruder FGSM ones. 这是一个关于 threat model 的重要提醒：在数字世界里”刚好越过边界”的扰动，在物理世界里可能一碰就碎。

10.2 Eyeglass frames and stop signs

• Sharif et al. (2016), “Accessorize to a Crime”（Spring 2022 的阅读材料，也是那张”戴古怪眼镜的人被认成明星”的 slide）：攻击者戴一副打印出来的眼镜框，让人脸识别系统认不出 TA（dodging）或者把 TA 认成指定的另一个人（impersonation）。眼镜框在对齐后的人脸图像里约占 6.5% 的像素。为了在物理世界可行，他们在优化中加入了额外约束：颜色必须是打印机能打出来的（non-printability score）、图案要平滑（total variation，既利于打印也更不显眼）、对姿态和光照的变化要鲁棒。扰动不再”小”，而是局部的、看得见的、但看起来像普通配饰。
• Eykholt et al. (2018), “Robust Physical-World Attacks on Deep Learning Visual Classification”：在真实的停车标志上贴黑白贴纸（看起来像涂鸦），让路标分类器把它认成限速标志。他们的 RP₂ 方法在实验室里让 100% 的图像被定向误分类，在行驶车辆上拍的视频帧里是 84.8%。

10.3 Expectation over Transformation

Athalye, Engstrom, Ilyas & Kwok (2018), “Synthesizing Robust Adversarial Examples”, made the physical recipe explicit. Let $\mathcal{T}$ be a distribution of transformations the input will undergo (viewpoint, scale, lighting, camera noise, 3D rendering). Optimize the expected loss:

and estimate its gradient by sampling: $\nabla_\delta \approx \frac1n\sum_{i=1}^n \nabla_\delta\,\ell(f(T_i(x + \delta)), y)$, which is unbiased as long as each $T_i$ is differentiable (or approximated by a differentiable surrogate). With EOT they 3D-printed a turtle that a standard ImageNet classifier labeled “rifle” from most viewpoints. EOT is also the standard tool for attacking randomized defenses (Lecture 8): if the defense adds randomness, average the gradient over it.

10.4 我们的”相机”（代码 04 演示 1）

变换 = 平移 ±1 像素、失焦（与 3 × 3 均值模糊按随机比例混合）、对比度 × U(0.8, 1.2)、传感器噪声 N(0, 0.05²)。每张对抗图随机变换 20 次，统计仍被分错的比例。噪声地板：干净图在同样的变换下也有 5.5% 被分错。前 300 张测试图：

• 最小扰动在物理世界里很脆弱：DeepFool 和 C&W（κ = 0）在数字世界里 99% 成功，变换后只剩 27–29%，能在 ≥ 90% 的变换下都成功的只有 1%。这正是 Kurakin et al. 观察到的现象的放大版。
• “深”比”巧”重要：C&W 加上 κ = 10，$L_2$ 翻了一倍多，变换后仍有 93.4%。
• EOT 用数字世界的成功率换变换后的成功率：ε = 0.08 时 PGD 0.920 → 0.641，EOT-PGD 0.787 → 0.748；”≥ 90% 次数仍成功”从 0.333 提高到 0.487（ε = 0.1：0.583 → 0.797）。代价是 8 倍的梯度计算。ε = 0.05 时 EOT 也只到 0.368：预算不够时，“对所有变换都有效”的扰动可能根本不存在。
• EOT 的收益比我预想的小，因为用 CE 的 PGD 本来就把样本推过边界很远（见代码 README）。

10.5 贴纸（代码 04 演示 2）

只允许改一块 4 × 4 的区域（16 个像素 = 6.25%，和眼镜框的比例差不多），区域内可以取 [0, 1] 里任意值；贴纸随物体一起平移。冒充的目标是 $t = (y + 3) \bmod 10$。前 200 张测试图：

• 一块贴在左上角背景上、大多数时候不碰数字的贴纸，能让约一半的图被认错 —— 模型在用数字以外的地方做决定。
• 冒充（定向）比躲避（非定向）难得多，和 §5.4 一致。
• 代码输出里冒充成功的例子：贴纸在一个 “1” 的左边画出了一横和一竖，看起来就是一个 “4”。压在物体上的贴纸可以直接改变语义。所以真实的物理攻击（眼镜框、“涂鸦”）还有一个 $L_p$ 球里没有的约束：贴纸本身要看起来无害。

11. Not bugs, features · 非鲁棒特征

11.1 The claim

Ilyas, Santurkar, Tsipras, Engstrom, Tran & Madry (2019), the subject of the Spring 2022 fireside chat, argued that adversarial vulnerability is not a bug of the model but a property of the data: datasets contain features that are highly predictive yet brittle and incomprehensible to humans, and standard training uses any feature that helps. Their definitions, for a binary label $y \in \{\pm1\}$ and a feature $f: \mathcal{X} \to \mathbb{R}$ normalized to mean 0 and variance 1:

• $\rho$-useful: $\mathbb{E}[y\cdot f(x)] \ge \rho$ — the feature is correlated with the label.
• $\gamma$-robustly useful: $\mathbb{E}\big[\inf_{\delta \in \Delta(x)} y\cdot f(x + \delta)\big] \ge \gamma$ — it stays useful under the worst allowed perturbation.
• useful, non-robust: $\rho$-useful for some $\rho$ bounded away from 0, but not $\gamma$-robustly useful for any $\gamma \ge 0$ — an adversary can flip its correlation with the label.

Under standard training (minimizing the classification loss), nothing distinguishes a robust feature from a non-robust one except how useful it is: any $\rho$-useful feature will be used. An adversarial perturbation, in this view, is simply the cheapest way to switch on the non-robust features of another class.

11.2 Their experiments

1. Disentangling. Using an adversarially trained (robust) model, they synthesized a “robustified” training set $\hat D_R$ whose images only carry the features that model uses; standard training on $\hat D_R$ gave a model with non-trivial robust accuracy. The same construction with a standard model ($\hat D_{NR}$) gave good clean accuracy but almost no robustness.
2. Non-robust features suffice. For each training pair $(x, y)$, pick a target $t$ — uniformly at random ($D_{\text{rand}}$) or deterministically $t = y + 1$ ($D_{\text{det}}$) — and replace $x$ by a small targeted adversarial perturbation $x_{\text{adv}}$ that a standard model classifies as $t$; relabel it $t$. To a human, every image looks mislabeled. In $D_{\text{det}}$ the robust features (what humans see) even point to the wrong class. Yet standard training on these sets generalizes to the original, correctly labeled test set:

3. Transferability. Architectures that learned more from $D_{y+1}$ were also more susceptible to transfer attacks from a ResNet-50 — consistent with “adversarial examples transfer because models pick up similar non-robust features” (§9.1).

11.3 一个知道真相的世界（代码 05 演示 1）

我们自己放一个”有用但不鲁棒”的特征进去，看会发生什么。每个类别配一个固定的淡斑点纹理（一半像素亮 0.08，其余为 0），加到这一类的每张图上；为了让形状本身不够用，每张图还随机遮挡一块 6 × 6。诊断用两个测试集：张冠李戴（形状是 $y$、纹理是 $y + 1$ 的）和纹理探针（只有纹理和噪声、没有数字）。

• 有用的、不鲁棒的特征会被使用，并且提高准确率：CNN 0.787 → 0.892，MLP 0.633 → 0.994。代价是鲁棒性：ε = 0.02 时 CNN 的鲁棒准确率从 0.629 掉到 0.197。标准训练只看”有没有用”。
• 用多少取决于架构：CNN 主要还跟着形状走（张冠李戴时 64.7% 跟形状），纹理只在形状被遮挡、不确定时帮忙；MLP 几乎完全靠纹理（94.8% 跟纹理，去掉纹理只剩 0.152）。“模型用数据里的哪些特征”不只是数据的性质，也是归纳偏置的性质 —— 这一点在 Ilyas et al. 的表述里不太显眼，在玩具世界里非常明显。（第一版设计没有遮挡，形状 100% 可预测，CNN 完全不用纹理；见代码 README。）
• “不鲁棒”是相对于 ε 的：两类纹理相差 0.08，ε = 0.02 还不够改写它，所以靠纹理的 MLP 在 ε = 0.02 时反而比只看形状的 MLP 更稳（0.482 vs 0.149）；ε = 0.04 时就几乎全被攻破（0.002）。

11.4 我们的 $D_{\text{rand}}$ / $D_{\text{det}}$（代码 05 演示 2–3）

主世界（没有故意放任何纹理），源模型是 Model A，定向 PGD-10，ε = 0.15（定向攻击需要大得多的预算，§5.4；ε = 0.03–0.05 时只有 3–13% 命中目标，$D_{\text{det}}$ 完全失败）。新模型和 A 同结构，只在 $(x_{\text{adv}}, t)$ 上训练，在原始的、标签正确的测试集上评估：

用”看起来全是错标”的数据训练，新模型在正确标注的测试集上远高于随机的 0.1；$D_{\text{det}}$ 里只有 18.3% 的预测是”形状 $y$ → 标签 $y + 1$“。定性上和论文的 CIFAR-10 结果一致（数值上 $D_{\text{det}}$ 的 0.435 与论文的 43.7% 接近纯属巧合，两个任务完全不同）。新模型毫无鲁棒性：它学到的正是那些不鲁棒的东西。

但扰动里到底是什么？ 七段数码管让我们可以检查：把 $D_{\text{det}}$ 的扰动 $\delta$ 和”真正把 $y$ 改成 $t$ 的笔画编辑”（同一字体下 $t$ 的字形 − $y$ 的字形）比方向。余弦相似度 0.242，而换成不相干的字体只有 0.043；$\delta$ 的能量有 9.4% 落在两个字形不同的那 5.8% 的像素上。所以扰动里确实有一点”淡淡的目标笔画”，但大部分能量在别处。

This is exactly the point of contention after the paper. In the Distill discussion (Engstrom et al. 2019, six commentaries with the authors’ responses), Gabriel Goh argued that part of what generalizes from $D_{\text{rand}}$ could be “robust feature leakage” — faint copies of real, robust features of $t$ inside the perturbation. He found that leakage alone buys only a small amount of test accuracy on CIFAR-10 $D_{\text{rand}}$, and the final version of the paper cites this; $D_{\text{det}}$ was designed so that robust features point to the wrong class, which makes leakage a weaker explanation there. Our toy cannot cleanly separate the two (an attempt to split δ into “on the glyph difference” and “elsewhere” was confounded by a label leak in the mask itself — see the code README).

有纹理的世界里，我们知道真相（代码 05 演示 3，$D_{\text{rand}}$，ε = 0.05，新模型都是 CNN）：

当源模型依赖纹理时，它的对抗扰动就是”把纹理换成目标类的”（cos 0.608）；一个从没见过任何正确标注的新 CNN，从这些扰动里学会了认纹理（探针 0.586，随机猜 0.1）。对抗扰动反映的是源模型用了什么特征，而这些特征可以被另一个模型学走 —— 这就是迁移性的一种解释。

11.5 Implications

• If vulnerability comes from the data, better fitting won’t remove it; robustness requires telling the model which features we want — e.g., by training against the adversary (Lecture 8). Ilyas et al. frame this as aligning the model’s features with human priors.
• Robust models’ gradients look more perceptually meaningful (Tsipras et al. 2019): robustness and interpretability (Lectures 2–3) are linked — a saliency map of a model that uses invisible features will look like noise, and that is a faithful explanation.
• “Non-robust” is always relative to a threat model: our texture was non-robust at ε = 0.04 but robust at ε = 0.02.

12. NLP 中的鲁棒性 · Robustness for text

12.1 为什么文本不一样

文本是离散的：没有”把一个词改 0.03”这回事，梯度给出的方向通常落不到一个真实的词上；而且”不改变意思”不是一个范数能定义的 —— “The movie was not bad” 和 “The movie was bad” 只差一个词，意思相反。所以 NLP 里的扰动集合是一组语义保持的改写：

判断一个改写”保持语义”需要人或另一个模型；这让文本攻击的评估比图像难得多。

12.2 SEARs

Ribeiro, Singh & Guestrin (2018), “Semantically Equivalent Adversarial Rules for Debugging NLP Models” (a Fall 2023 Lecture 6 reading), proceed in two steps. First, find semantically equivalent adversaries (SEAs) for individual inputs: generate paraphrases (via back-translation), keep those judged equivalent, and pick ones that change the prediction. Second, generalize them into simple replacement rules that induce adversaries on many instances — rules of the flavor “What is → What’s” or “color → colour”. Across reading comprehension, visual question answering and sentiment analysis, their rules induced about four times as many mistakes as the bugs found by human experts, and augmenting the training data with the rules reduced the errors while maintaining accuracy. 规则的价值在于它是可读的 bug 报告：一条规则告诉你模型在哪一类输入上不稳定，而不只是一个孤立的反例。

12.3 CheckList

Ribeiro, Wu, Guestrin & Singh (2020), “Beyond Accuracy: Behavioral Testing of NLP Models with CheckList” (ACL 2020 best paper), borrow from software testing: a matrix of linguistic capabilities (vocabulary, negation, named entities, robustness to typos, …) × test types:

• MFT (Minimum Functionality Test)：小而简单、标签明确的句子，像单元测试（“I didn’t love the food” 应为负面）。
• INV (Invariance test)：做一个不该改变预测的修改（换人名、换地名、加拼写错误），检查预测不变。
• DIR (Directional expectation test)：做一个应该让预测往某个方向变的修改（加一句负面评价），检查方向。

Tests are generated from templates, so they need no labeled data. CheckList found serious failures in commercial sentiment models and in state-of-the-art research models; in a user study, practitioners using it created twice as many tests and found almost three times as many bugs.

我们的玩具版（代码 04 演示 3）：一个词袋（bag-of-words）逻辑回归情感模型，训练语料里故意放了两个伪相关（正面影评更常说 “film”、更常提到演员 chen）。测试集准确率 0.942：

0.942 的准确率背后：模型完全不懂否定（看到 “good” 就说正面，“not” 的权重是 +0.39），词表外的同义词被直接忽略，换个人名就有 11% 的影评改变预测（chen 的权重 +1.11），一条无害的改写规则翻转 7%（film +0.84、movie −0.95）。DIR 测试 0 失败 —— 词袋模型是单调的，这个测试它天然通过。行为测试的价值在于覆盖不同的能力：一个漂亮的总准确率可以藏住整类的失败（第 1 讲 §2.1 的”平均数”问题）。

12.4 同义词替换攻击（代码 04 演示 4）

贪心地把情感词换成同义词表里的词（意思基本不变），每次挑让 P(真实标签) 降得最多的替换：933 条可攻击的影评中 37.3% 被翻转；成功时中位数只换 2 个词，46.6% 只换 1 个。例子：

• 原句（P(正面) = 0.991）：the cast was quite wonderful and i thought the acting was great
• 改写（P(正面) = 0.492）：the cast was quite fantastic and i thought the acting was fantastic

“fantastic” 不在训练词表里，证据直接消失，模型只能靠 “cast”、“acting” 这些中性词的偶然权重。真实的 Transformer 不会这么天真，但同义词、拼写、人名、否定正是 CheckList 在商用模型上发现 bug 的那几类。

12.5 From classifiers to LLM applications

Fall 2023 Lecture 6 lists robustness as one of four trust checks for any “LLM for X” project and gives concrete probes: for a doctor-note summarizer, robustness to “seemingly benign perturbations” (adding / dropping punctuation and stop words) and to the order of information (prescriptions mentioned before the diagnosis); for homework grading, whether a student could slightly tweak an incorrect solution and have it accepted. It also lists a second goal — making the model generate bad text (Kang et al. 2023, “Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks”) — and mitigations (input filters, output filters, RLHF). 这些就是 CheckList 的 INV / DIR 测试在生成式应用里的样子，也是第 15 讲的起点。

13. 桥梁 · Bridges

13.1 To distribution shift（第 1 讲）

Adversarial examples and distribution shift are two answers to “what if the inputs move?” (Lecture 1 §2.2):

两者相关但不等价：对 $L_\infty$ 鲁棒的模型不一定对自然损坏鲁棒，反之亦然；物理世界攻击（§10）正好介于两者之间 —— 它是在一个变换分布上找最坏的扰动。第 1 讲的 officer_flag 和本讲的纹理有一个共同点：模型依赖了一个在训练分布上有用、换一个分布（或换一个对手）就失效的特征。

13.2 To LLM jailbreaks（第 15 讲）

The attacks of this lecture reappear almost one-to-one for aligned LLMs. GCG (Zou et al. 2023, “Universal and Transferable Adversarial Attacks on Aligned Language Models”) searches for an adversarial suffix of tokens that makes the model begin its answer with an affirmative phrase. Because tokens are discrete, it uses gradients with respect to one-hot token indicators to propose candidate swaps and then evaluates them — the HotFlip idea of §12.1 combined with greedy coordinate search, i.e. a discrete cousin of PGD. Its suffixes transfer to other models, including black-box commercial ones (§9). Prompt injection is a new threat model (who controls which part of the input) rather than a new optimizer.

13.3 To defenses（第 8 讲）

Everything here sets up Lecture 8: adversarial training plugs PGD into the training loop (Madry’s min–max, §5.3); certification turns the upper bounds of §8 into lower bounds; and §6.6 is the first instance of obfuscated gradients, the reason most early defenses were broken. 第 8 讲还会讨论鲁棒性与准确率的冲突（Tsipras et al. 2019）—— 本讲 §11.3 的表已经给了一个小例子：去掉纹理（只用鲁棒特征）的 CNN 准确率低了 10 个百分点，ε = 0.02 时鲁棒准确率却高 3 倍。

14. 小结 · Summary

Five takeaways:

1. Robustness is defined by a threat model, and the threat model must be written down before any number is reported: norm, budget, knowledge, and what counts as “the same input”.
2. Random-noise testing badly overestimates robustness. Worst-case directions exist because many small, aligned changes add up in high dimensions (Lecture 1 §5.3, now on a CNN).
3. An attack is an upper bound on robustness, and its quality is the whole game. Use strong, well-tuned, diverse attacks (PGD with restarts, a margin loss, a gradient-free attack, transfer); the defensive-distillation story is what happens otherwise.
4. Secrecy and the digital/physical gap are not defenses. Adversarial examples transfer, gradients can be estimated from queries, and EOT makes perturbations survive the camera.
5. Adversarial examples tell us what models use. Non-robust features are real and predictive; models pick them up depending on their inductive bias; and the same question — “what does the model rely on?” — connects robustness to explanations (L2–L4) and to LLM jailbreaks (L15).

15. 常见误区 · Common pitfalls

• “We added random noise and accuracy didn’t change, so the model is robust.” ε = 0.1 random noise: 0.988; PGD at the same ε: 0.012–0.024 (§4.2).
• “Our attack found no adversarial examples, so the model is robust.” A failed attack is not a proof. Check that larger ε reaches 100%, that PGD beats FGSM, and that a margin loss or a black-box attack doesn’t do better (§8.2). The “logit × 100” model was perfectly robust to FGSM and not at all robust (§6.6).
• “Robust at $L_\infty$ ε = 8/255” means robust. Only against that threat model. $L_2$, $L_0$, patches, transformations and semantic edits are different sets (§2.2, §5.5, §10.5).
• “Our model is private, so attackers need white-box access.” Transfer from a self-trained surrogate (70–85% here) and query-based estimation (90.5%, with a median of 766 queries) both work without it (§9).
• “The smallest perturbation is the most dangerous.” In the physical world the opposite holds: minimal perturbations died under camera transforms (27–29%), high-confidence ones survived (93%) (§10.4).
• “Adversarial perturbations are meaningless noise.” They often point toward a semantically neighboring class (5→6, 8→0) and can carry real, learnable features — a new model trained only on them reached 0.435 on the clean test set (§5.4, §11.4).
• “Targeted and untargeted success rates are comparable.” At ε = 0.05 here: 0.123 vs 0.510 (§5.4).
• “High accuracy on the test set means the NLP model is fine.” 0.942 accuracy with 100% failure on negation (§12.3).

16. 练习 · Exercises

练习 1（对偶范数与 FGSM）. (a) 证明 $\max_{\lVert\delta\rVert_\infty \le \epsilon} g^\top\delta = \epsilon\lVert g\rVert_1$，并写出最优的 $\delta$。(b) 对 $\lVert\delta\rVert_1 \le \epsilon$ 做同样的事；这对应哪一类攻击？(c) 有 [0, 1] 盒约束时，对一个像素值恰好为 0、梯度为负的像素，FGSM 的扰动实际是多少？这解释了 §3.2 的什么现象？

(a) $g^\top\delta = \sum_i g_i\delta_i \le \sum_i |g_i|\,|\delta_i| \le \epsilon\sum_i|g_i|$，在 $\delta_i = \epsilon\,\text{sign}(g_i)$ 时取等号。(b) $\sum_i g_i\delta_i \le \max_i|g_i|\cdot\sum_i|\delta_i| \le \epsilon\lVert g\rVert_\infty$，最优解把全部预算放在 $|g_i|$ 最大的那个坐标上 —— 稀疏的、“只改一个像素”的攻击，是 $L_0$ / JSMA 类攻击的线性版本。(c) FGSM 想把它减小 ε，但裁剪到 [0, 1] 后实际改动为 0。黑背景上约一半的像素恰好是 0，落在这些像素上的随机 −ε 也会被这样吃掉，所以随机扰动的有效大小比名义上的小，而 FGSM 只会在背景上做”+ε”的改动（演示 1 的字符画里背景大多是 ’+’）。这是 §3.2 中随机/对抗的比值（约 5）小于 $\sqrt{2d/\pi} \approx 12.8$ 的原因之一。

练习 2（PGD 的投影）. (a) 写出把 $\delta$ 投影到 $\{\lVert\delta\rVert_2 \le \epsilon\}$ 的公式，并证明它确实是欧氏距离下最近的点。(b) 为什么”先投影到 ε 球、再裁剪到 [0, 1]“得到的点仍然在 ε 球里？(c) 它是否等于”投影到 ε 球与 [0, 1]ᵈ 的交集”？如果不是，为什么实践中仍然可以接受？

(a) $\Pi(\delta) = \delta\cdot\min(1, \epsilon/\lVert\delta\rVert_2)$。若 $\lVert\delta\rVert_2 \le \epsilon$ 则不动；否则球内离 $\delta$ 最近的点在 $\delta$ 方向的射线上（对任意 $u$ 在球内，$\lVert\delta - u\rVert \ge \lVert\delta\rVert - \lVert u\rVert \ge \lVert\delta\rVert - \epsilon$，等号在 $u = \epsilon\delta/\lVert\delta\rVert$ 时成立）。(b) $x_i \in [0, 1]$，裁剪 $x_i + \delta_i$ 到 [0, 1] 只会让 $|\delta_i|$ 变小，所以任何 $L_p$ 范数都不会变大。(c) 对 $L_\infty$，两个约束都是逐坐标的区间，先后裁剪就是交集上的精确投影；对 $L_2$ 一般不相等（精确投影需要解一个小的一维问题）。实践中 PGD 只需要每一步保持在可行集内并且大致朝上升方向走，“近似投影”不影响它作为攻击的有效性。

练习 3（C&W 的设计）. (a) 对 untargeted C&W，说明为什么 $f(x') = \max(m(x', y), -\kappa)$ 的全局最优（在 $c$ 合适时）落在 $m = -\kappa$ 附近，而把 $f$ 换成 $-\text{CE}(x', y) = \log p_y$ 时不会。(b) tanh 换元下，$x'_i$ 对 $w_i$ 的导数是什么？当 $x'_i$ 接近 0 或 1 时它有什么问题？与”直接裁剪”相比，问题是更好还是更差？(c) 用 §6.4 的表解释为什么需要对每个样本二分搜索 $c$。

(a) 当 $m > -\kappa$ 时，$f = m$，减小 $m$ 会减小目标函数；一旦 $m \le -\kappa$，$f$ 恒为 $-\kappa$，梯度为 0，剩下的只有 $\lVert\delta\rVert^2$ 项把 $x'$ 拉回 $x$ —— 于是最优点停在 $m = -\kappa$ 的边界上（如果 $c$ 大到足以让越过边界值得）。$\log p_y$ 没有下界，越过边界之后仍然在下降，它会和 $\lVert\delta\rVert^2$ 一直”拔河”，最优点停在边界之外的某处，扰动不是最小的；而且 $p_y \approx 1$ 时它的梯度被 softmax 压扁（§6.5 的表）。(b) $\partial x'_i/\partial w_i = \frac12(1 - \tanh^2 w_i) = 2x'_i(1 - x'_i)$，接近边界时趋于 0，梯度变小但不为 0，而且 Adam 的自适应步长会部分补偿；直接裁剪则在边界外梯度恰好为 0、优化停在原地。(c) c = 0.03 时 42% 成功，c = 0.1 时 99% 成功、c = 100 时扰动是两倍大：每个样本”刚好够用”的 c 不同，固定一个 c 要么让一些样本失败，要么让另一些扰动不必要地大。

练习 4（查询代价）. 一个黑盒 API 对 $d = 150{,}528$（224 × 224 × 3）的图像返回概率。(a) 用逐坐标中心差分估计一次完整梯度需要多少次查询？(b) NES 用 $n = 50$ 次查询估计梯度。设真实梯度为 $g$，$u \sim \mathcal{N}(0, I)$。证明 $\mathbb{E}[(g^\top u)u] = g$，并说明为什么只用 50 个方向、估计非常粗糙时，PGD 的 sign 步仍然有用。(c) 代码 03 演示 4 中，NES 成功样本的查询中位数是 766。如果 API 每次查询收费 0.001 美元、每张图最多允许 1000 次查询，攻击 100 张图大约花多少钱、成功多少张？

(a) $2d = 301{,}056$ 次。(b) $\mathbb{E}[(g^\top u)u] = \mathbb{E}[uu^\top]g = Ig = g$（一阶近似下 $[\ell(x + \sigma u) - \ell(x - \sigma u)]/(2\sigma) \approx g^\top u$）。估计 $\hat g$ 与 $g$ 的夹角可能很大，但只要 $\hat g$ 与 $g$ 正相关，sign$(\hat g)$ 的每一步在期望上就是上升方向；PGD 走很多小步，误差会平均掉 —— 就像 SGD 用很嘈杂的梯度也能收敛。(c) 预算 1000 时成功率 0.605，约 60 张成功；成功的样本平均花不到 1000 次查询，失败的花满 1000 次，总花费不超过 100 × 1000 × 0.001 = 100 美元。对一个攻击者来说很便宜 —— 限速和计费能提高代价，但不能阻止攻击。

练习 5（EOT 的估计）. (a) 说明 $\frac1n\sum_i\nabla_\delta\ell(f(T_i(x + \delta)), y)$ 是 $\nabla_\delta\mathbb{E}_T[\ell(f(T(x + \delta)), y)]$ 的无偏估计需要什么条件。(b) 我们的 random_transform 里有 clamp(0, 1)；为什么直接用 clamp 会让 EOT 失效，代码是怎样修的？这个修法在数学上是在估计什么？(c) 用 §10.4 的表解释：为什么 ε = 0.05 时 EOT 的”变换后成功率”也只有 0.368？

(a) 每个 $T$ 对输入（几乎处处）可导，并且可以交换期望与求导（例如 loss 和它的梯度有界、$T$ 的分布不依赖 $\delta$）；$T_i$ 是独立同分布的采样。(b) PyTorch 中 clamp 对恰好等于边界的值传回 0 梯度，而背景里大约一半的像素恰好为 0，所以 EOT 永远不会改变这些像素（sign(0) = 0）。代码改用 straight-through：前向截断，反向当作恒等函数。数学上它估计的不再是”截断后函数”的精确梯度，而是一个”把截断当成恒等”的替代梯度 —— 和第 8 讲 BPDA（Backward Pass Differentiable Approximation）的思路相同：用一个可导的近似替代不可导（或梯度没用）的部件。(c) 对整个变换分布都有效的扰动需要更大的”深度”：平移、模糊会稀释细粒度的扰动，对比度和噪声会移动边界附近的点。ε = 0.05 时，即使只对未变换的图，PGD 也只有 52% 成功；要求对 20 次随机变换中的大多数都成功，在这个预算下多数样本上根本不存在这样的扰动。EOT 是一个更难的优化问题的正确解法，它不能凭空制造预算。

练习 6（非鲁棒特征的思想实验）. 考虑一个简单的高斯玩具模型：$y \in \{\pm1\}$ 等概率；特征 $x_1 = y + \mathcal{N}(0, 1)$（鲁棒、信号强度 1）；特征 $x_2, \dots, x_{d+1}$ 都是 $\eta y + \mathcal{N}(0, 1)$（每个都很弱，$\eta = 0.1$）。(a) 只用 $x_1$ 的最优线性分类器的准确率是多少？(b) 用全部特征、权重 $w = (1, \eta, \dots, \eta)$ 时，信噪比（均值差 / 标准差）是多少？$d = 400$ 时准确率约多少？(c) 一个 $L_\infty$ 对手可以把每个特征改 $\epsilon = 2\eta = 0.2$。它能对 (b) 的分类器做什么？对 (a) 的呢？(d) 这和 §11.3 的表有什么对应？

(a) 判 sign$(x_1)$，准确率 $\Phi(1) \approx 0.841$。(b) $w^\top x$ 的条件均值 $y(1 + d\eta^2)$，标准差 $\sqrt{1 + d\eta^2}$，信噪比 $\sqrt{1 + d\eta^2}$；$d = 400$ 时 $\sqrt{5} \approx 2.24$，准确率 $\Phi(2.24) \approx 0.987$。弱特征合起来非常”有用”。(c) 对手把每个弱特征移动 $-2\eta y$，它们的均值从 $\eta y$ 变成 $-\eta y$，加权和的均值变成 $-d\eta^2 y = -4y$；$x_1$ 也被移动 $-0.2y$，均值 $0.8y$。于是 $w^\top x$ 的条件均值是 $(0.8 - 4)y = -3.2y$，标准差仍是 $\sqrt5 \approx 2.24$，准确率约 $\Phi(-1.43) \approx 0.08$ —— 分类器几乎总是判错。对 (a)，对手只能把 $x_1$ 的均值从 $y$ 移到 $0.8y$，准确率 $\Phi(0.8) \approx 0.79$，几乎不受影响。这就是 Tsipras et al.（2019）和 Ilyas et al. 的理论模型的简化版：用弱特征换来准确率，也换来脆弱性。(d) 纹理就是”很多个弱特征”：它让准确率从 0.787 升到 0.892（CNN）、0.633 升到 0.994（MLP），也让 ε = 0.04 的鲁棒准确率掉到接近 0；而”不鲁棒”取决于 ε 与特征强度的比较 —— 在 (c) 中如果 $\epsilon < \eta$，弱特征的符号翻不过来，这对应 MLP 在 ε = 0.02 时反而更稳。

代码相关的练习（“动手改一改”）见 code/lecture07/README.md。

17. 延伸阅读 · Further reading

本讲的主要来源

• CS329T Spring 2022 robustness 模块 slides（Datta & Mitchell）与 HW5（PGD、C&W L2、迁移；作业说明为 Spring 2021 版）；CMU 18-739 中 Carlini 讲 “Towards Evaluating the Robustness of Neural Networks” 的 slides；CS329T Fall 2023 Lecture 6 “Project Design and Planning”（Taly、Mitchell、Datta）。
• Szegedy et al., “Intriguing Properties of Neural Networks”, ICLR 2014；Goodfellow, Shlens & Szegedy, “Explaining and Harnessing Adversarial Examples”, ICLR 2015。

白盒攻击

• Kurakin, Goodfellow & Bengio, “Adversarial Examples in the Physical World”, ICLR Workshop 2017（BIM、打印拍照实验）；Kurakin, Goodfellow & Bengio, “Adversarial Machine Learning at Scale”, ICLR 2017。
• Madry, Makelov, Schmidt, Tsipras & Vladu, “Towards Deep Learning Models Resistant to Adversarial Attacks”, ICLR 2018。
• Carlini & Wagner, “Towards Evaluating the Robustness of Neural Networks”, IEEE S&P 2017；Carlini & Wagner, “Defensive Distillation is Not Robust to Adversarial Examples”, 2016；Papernot et al., “Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks”, IEEE S&P 2016。
• Papernot et al., “The Limitations of Deep Learning in Adversarial Settings”, EuroS&P 2016（JSMA）；Moosavi-Dezfooli, Fawzi & Frossard, “DeepFool”, CVPR 2016。
• Carlini et al., “On Evaluating Adversarial Robustness”, 2019；Croce & Hein, “Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks”, ICML 2020（AutoAttack）。

迁移与黑盒

• Papernot, McDaniel & Goodfellow, “Transferability in Machine Learning”, 2016；Papernot et al., “Practical Black-Box Attacks against Machine Learning”, AsiaCCS 2017；Liu, Chen, Liu & Song, “Delving into Transferable Adversarial Examples and Black-box Attacks”, ICLR 2017。
• Chen et al., “ZOO”, AISec 2017；Ilyas, Engstrom, Athalye & Lin, “Black-box Adversarial Attacks with Limited Queries and Information”, ICML 2018；Uesato et al., “Adversarial Risk and the Dangers of Evaluating Against Weak Attacks”, ICML 2018；Brendel, Rauber & Bethge, “Decision-Based Adversarial Attacks”, ICLR 2018。

物理世界

• Sharif, Bhagavatula, Bauer & Reiter, “Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition”, CCS 2016；Eykholt et al., “Robust Physical-World Attacks on Deep Learning Visual Classification”, CVPR 2018；Athalye, Engstrom, Ilyas & Kwok, “Synthesizing Robust Adversarial Examples”, ICML 2018。

为什么存在

• Ilyas et al., “Adversarial Examples Are Not Bugs, They Are Features”, NeurIPS 2019；Engstrom et al., “A Discussion of ‘Adversarial Examples Are Not Bugs, They Are Features’”, Distill 2019；Tsipras et al., “Robustness May Be at Odds with Accuracy”, ICLR 2019。
• Gilmer et al., “Adversarial Spheres”, 2018；Shafahi et al., “Are Adversarial Examples Inevitable?”, ICLR 2019；Tanay & Griffin, “A Boundary Tilting Perspective on the Phenomenon of Adversarial Examples”, 2016；Stutz, Hein & Schiele, “Disentangling Adversarial Robustness and Generalization”, CVPR 2019。

NLP

• Ribeiro, Singh & Guestrin, “Semantically Equivalent Adversarial Rules for Debugging NLP Models”, ACL 2018；Ribeiro, Wu, Guestrin & Singh, “Beyond Accuracy: Behavioral Testing of NLP Models with CheckList”, ACL 2020。
• Ebrahimi et al., “HotFlip”, ACL 2018；Alzantot et al., “Generating Natural Language Adversarial Examples”, EMNLP 2018；Jin et al., “Is BERT Really Robust?”（TextFooler）, AAAI 2020；Jia & Liang, “Adversarial Examples for Evaluating Reading Comprehension Systems”, EMNLP 2017。
• Kang et al., “Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks”, 2023；Zou et al., “Universal and Transferable Adversarial Attacks on Aligned Language Models”, 2023（第 15 讲）。

分布漂移

• Hendrycks & Dietterich, “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations”, ICLR 2019。

18. 下一讲预告 · Next lecture

Lecture 8：防御与可证明鲁棒性（Defenses & Certified Robustness）。

本讲的每一个结论都是”攻击找到了”—— 鲁棒准确率的上界。第 8 讲问反过来的问题：怎样让模型真的更鲁棒，怎样证明一个预测在某个扰动集合里不会改变？我们会把本讲的 PGD 放进训练循环（Madry 的 min–max adversarial training），看它的代价（干净准确率下降，Tsipras et al. 的 “robustness may be at odds with accuracy”，本讲 §11.3 已经有一个小例子）；回顾为什么很多早期防御被攻破（本讲 §6.6 的防御性蒸馏是第一个 obfuscated gradients 的例子，第 8 讲会系统地讲 BPDA 和 EOT 怎样绕过它们）；然后从”测试”走到”证明”：Lipschitz 约束与 margin（GloRo）、局部鲁棒性的几何投影、randomized smoothing 的认证半径 —— 也就是第 1 讲 §7.3 “能证明的就证明，其余的弃权”在深度网络上的版本。最后，解释本身也可以被攻击：一个解释看起来无害、行为却有歧视的模型。