Lecture 9 · 隐私攻击 · Privacy Attacks on Machine Learning

对应 Maps to：Spring 2022 Week 8 “Privacy, Week 1 (only week)“（Anupam Datta & John Mitchell：隐私风险的三个来源、data linking —— Sweeney 找出州长 Weld 的病历、“87% 的美国人可被 5 位邮编 + 性别 + 出生日期唯一确定”、Netflix–IMDb 去匿名化、differential privacy 的想法、membership inference（Shokri 的 slides）；阅读材料：Narayanan & Shmatikov 2008 “Robust De-anonymization of Large Sparse Datasets”、Fredrikson et al. 2014 “Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing”；fireside chat：Sablayrolles et al. 2019 “White-box vs Black-box: Bayes Optimal Strategies for Membership Inference”）与 Week 9（membership inference 与过拟合；阅读：Shokri et al. 2017 “Membership Inference Attacks Against Machine Learning Models”、Leino & Fredrikson 2020 “Stolen Memories”）；HW4（Shokri 式黑盒影子模型攻击、过拟合与攻击成功率的关系；作业说明是 2021 年版，2022 年改为选做；其中 DP 防御的部分属于第 10 讲）；Fall 2023 课程大纲中的 “Explainability – influence functions and privacy” 一周（第 11 讲 influence functions 客座讲座见第 4 讲）；Fall 2024 Lecture 3 列出的 privacy、unlearning 与 copyright。Fall 2023 / 2024 这几部分没有公开的隐私 slides，相应内容按公开论文撰写（LiRA、训练数据提取、memorization 的规模效应、machine unlearning）。 配套代码 Code：code/lecture09/（numpy；5 个脚本 + 2 个工具模块；每个脚本 1–10 秒） 预计阅读时间：3 小时（含运行代码）

0. 本讲导航 · How to read this lecture

0.1 Learning goals

After this lecture you should be able to:

1. 说清楚一个 ML 系统的隐私风险从哪里来（存储的数据、数据访问、模型访问），以及攻击者可能推断出什么：身份（re-identification）、成员身份（membership）、属性（attribute inference / model inversion）、训练数据本身（extraction）。
2. Explain why removing names is not anonymization: quasi-identifiers, linkage, k-anonymity and its failure modes (homogeneity, high-dimensional sparse data), with numbers you can reproduce.
3. 区分”统计推断”和”训练数据的泄露”：用成员 vs 非成员的对照，判断一个 model inversion 攻击到底泄露了什么。
4. Frame membership inference as a hypothesis test, and explain why the field moved from accuracy / AUC to TPR at low FPR; derive why the likelihood ratio is optimal (Neyman–Pearson) and why a per-example threshold is what makes attacks strong (Sablayrolles et al.; LiRA).
5. Implement and compare the loss-threshold attack, Shokri’s shadow models, difficulty calibration and LiRA (offline / online), report them with a noise floor, and identify who is vulnerable (outliers, surprising labels, duplicates, minority subgroups).
6. 描述语言模型的记忆与训练数据提取（canary 与 exposure、GPT-2 提取、memorization 的三个规模规律），以及”不靠 DP”的防御（正则化、去重、只返回标签）和 machine unlearning 各自的局限，以及怎样用 MIA 去验证它们。

0.2 先修与本讲范围 Prerequisites & scope

0.3 本讲地图 Map

§1  隐私风险从哪里来；攻击者能推断出什么；"推断"不等于"泄露"
§2  去匿名化与链接攻击：准标识符、k-anonymity、稀疏高维数据             ← 代码 01
§3  Model inversion 与属性推断：成员 vs 非成员的对照                    ← 代码 02
§4  成员推断：一个假设检验；为什么要看"低 FPR 下的 TPR"；实验台          ← 代码 03 演示 1
§5  攻击：loss 阈值 → 影子模型 → Bayes 最优 → 白盒 → LiRA              ← 代码 03 演示 2–5
§6  怎样报告隐私风险（一张清单）
§7  谁会被认出来：离群、反常标签、重复、少数群体；隐私洋葱；λ 与 n      ← 代码 04 演示 1–3
§8  不靠 DP 的防御与 machine unlearning：label-only 攻击、SISA、遗忘的验证 ← 代码 04 演示 4–5
§9  语言模型：canary 与 exposure、训练数据提取、规模规律、版权            ← 代码 05
§10–14 小结、误区、练习、延伸阅读、下一讲预告

0.4 Notation

本讲按群体比较时直接报告每个群体的数值（不报告差值），所以不涉及第 1、5、6 讲的差值符号约定。

0.5 关于数字 · About the numbers

讲义里所有实验数字都来自配套代码的实际输出（固定随机种子，同一台机器上重复运行逐字相同）。信贷部分用第 1 讲的合成信贷世界（只用 5 个决策时拿得到的特征）；人口、医院数据、评分库、语料都是合成的。论文里的数字（87%、99%、604 条、1% 等）来自论文本身，并在文中注明。有几个结果和我原本的预期不同（例如 offline LiRA 在我们的玩具里很弱、正则化对 LiRA 的低 FPR 指标不是单调的），讲义按实际结果写，代码 README 的”关于实验结果”逐条记录。

1. 隐私风险从哪里来 · Where privacy risk comes from

1.1 三个来源

Spring 2022 隐私模块的第一张内容幻灯片把风险分成三类：

1. 存储的数据（data stored to build ML models）：数据库被泄露、被滥用。
2. 数据访问（data access provided to the data science team）：分析师、外包方、“匿名化”之后公开发布的数据集。
3. 模型访问（access to ML models built from data with sensitive elements）：模型本身是训练数据的函数，会携带训练数据的信息 —— 本讲的大部分内容。

课上举的伤害也很具体：冒用与盗窃（拿到你的密码、卡号、身份信息后改地址、开新账户）和不公平的对待（根据私人信息歧视、以”疑似既往病史”拒保、按与工作无关的私人信息拒聘）。还有一条朴素但最有效的原则：不要存你用不到的信息（Do not store information you will not need）—— 删掉不需要的字段、用随机 ID 代替姓名、用区间代替精确数值。幻灯片紧接着提醒：but beware: these methods do not usually add much privacy。§2 就是这句话的证明。

1.2 攻击者能推断出什么 · A taxonomy

这些攻击之间有很强的联系：extraction 用 membership inference 给候选文本排序（§9.2）；attribute inference 和 membership inference 在 Yeom et al.（2018）的框架里是两种密切相关的 advantage；而 membership inference 是”泄露”最基本的测量单位 —— 差分隐私（第 10 讲）恰好就是对”任何成员推断攻击的成功率”给出上界。

1.3 Threat model：攻击者能做什么

和第 7 讲的对抗攻击一样，任何隐私结论都要先说清楚威胁模型：

• Access：一份发布的数据表；黑盒 API 只返回标签；返回置信度向量；白盒（拿到参数、梯度）；能参与训练（投毒、在联邦学习里发送更新）。
• Auxiliary knowledge：攻击者知道的同分布数据、公开的人口统计、目标的部分属性、训练算法和模型结构。
• Goal：针对一个具体的人（targeted），还是从整体里尽可能多地识别（untargeted）；只要”有一部分人能被确定地识别”就算成功，还是要”平均猜对”。

最后一条决定了用什么指标，§4.3 会看到它的后果。

1.4 “推断”不等于”泄露” · Inference is not (necessarily) a breach

Spring 2022 的讨论题里有一个经典的例子（来自 Dwork 对差分隐私的解释）：side information: Terry Gross is two inches shorter than the average Lithuanian woman; data query: what is the average height of a Lithuanian woman? 发布”立陶宛女性的平均身高”这个统计量，就泄露了 Terry Gross 的身高 —— 不管她在不在数据库里。

这个例子说明一个关键区分：

• Population inference（统计推断）：模型（或统计量）学到了关于人群的规律，用它可以推断任何人的属性。这本来就是统计和机器学习的目的；如果禁止它，任何有用的数据分析都不可能。
• Individual leakage（个人信息的泄露）：模型的输出因为某个人参加了训练而改变，从而透露了只关于这个人的信息。

差分隐私（第 10 讲）选择保护后者：“有没有你，结果差不多”。所以判断一个攻击是不是”隐私泄露”，最干净的办法是对照：同样的攻击在训练集成员和非成员身上效果一样吗？§3 用这个对照重新审视 model inversion，§4–§7 的成员推断则直接测量后者。

2. 去匿名化与链接攻击 · Re-identification and linkage

代码 01 在一个合成人口上重做本节的每个实验：40 个五位邮编（每个 1571–31429 人）、共 433,078 人，每人有性别、出生日期和一个诊断。

2.1 Sweeney 与州长 Weld

1990 年代，马萨诸塞州团体保险委员会（Group Insurance Commission）为研究目的公布了州雇员的”已匿名化”医疗记录：删掉了姓名、地址、社保号，但保留了邮编、性别、出生日期。州长 William Weld 公开表示患者隐私受到了保护。当时还是研究生的 Latanya Sweeney 花 20 美元买了剑桥市的选民名册（含姓名、地址、邮编、性别、出生日期），把两份数据按这三列连接起来 —— 和 Weld 同一天出生的人只有 6 个，其中 3 个男性，只有 1 个住在他的邮编 —— 然后把州长本人的病历寄到了他的办公室。她后来估计，87% 的美国人可以被”5 位邮编 + 性别 + 出生日期”唯一确定（Sweeney 2000，基于 1990 年人口普查）；Golle（2006）用 2000 年普查数据重算，结果是 63%。

这三列叫 quasi-identifiers（准标识符）：每一列单独看都不敏感、也不能确定任何人，组合起来却几乎是身份证号。代码 01 演示 1：

为什么会这样？纯粹是计数。(性别, 出生日期) 大约有 $2 \times 73 \times 365 \approx 5.3$ 万种组合，而一个邮编只有几千到几万人。人数远少于”格子”数时，大多数格子里只有 0 或 1 个人。如果一个邮编有 $m$ 人、生日均匀分布在 $D$ 个格子里，某个人是唯一的概率约为 $(1 - 1/D)^{m-1} \approx e^{-m/D}$（练习 3）。所以邮编越小越危险：

（真实数据里也是农村比城市更容易被唯一识别。）更新的研究把这个结论推得更远：Rocher, Hendrickx & de Montjoye（2019）估计，用 15 个人口学属性，99.98% 的美国人可以在任何数据集里被正确地重新识别。

2.2 链接攻击的两个方向

演示 2 模拟 Sweeney 的设定：医院公布 2% 的人（8630 条记录，删掉姓名），选民名册包含 60% 的人（259,946 人，带姓名）。

两个方向要分开看：

• Target → record（Sweeney 找 Weld 的方式）：攻击者已经知道某人在数据里（州长是州雇员），只需在公布的数据里查 TA 的准标识符。只要组合在公布的数据内部唯一（0.991），就一定找到了 TA 的那一行 —— 不会错。
• Record → name：从一条病历出发，在选民名册里找唯一匹配。这需要本人在名册里、并且名册里没有别人恰好同属性。0.494 的病历被正确连上姓名；但唯一匹配里有 11.5% 是错的（本人不在名册里，名册里恰好有另一个同属性的人）—— 链接攻击也会冤枉人。

另外注意：把邮编粗化到三位远远不够（仍有 0.286 被正确识别），把出生日期粗化到年份则几乎消除了链接。“泛化哪一列”很重要 —— 这引出了 k-anonymity。

2.3 k-anonymity 及其失败 · k-anonymity and why it is not enough

Definition (Samarati & Sweeney 1998; Sweeney 2002). A released table is k-anonymous if every combination of quasi-identifier values that appears in it appears in at least $k$ records. Two operations achieve it: generalization (replace a value by a coarser one: ZIP 94305 → 943**, a date → its year) and suppression (drop the few records that would still be in tiny groups).

演示 3 在公布的 8665 条记录上比较两种算法。（a）全列泛化（full-domain generalization，Samarati & Sweeney 的做法：每个属性整列升到同一层级）：在 3 × 5 × 2 = 30 种方案里，选”删除 ≤ 1% 记录后满足 k-anonymity、平均组最小”的那个。（b）Mondrian（LeFevre, DeWitt & Ramakrishnan 2006）：递归地在最宽的维度上按中位数切分，人多的地方切得细、人少的地方切得粗。

三个教训：

1. Utility cost. 全列泛化很浪费：为了 k = 10 要把邮编整列去掉；k = 50 时平均每组 327 人，对任何按年龄或地区的分析都几乎没用了。Mondrian 按区域选粒度，同样满足 k = 5 只需平均 6.8 人一组。
2. Homogeneity attack. 组越接近 k，越可能出现”组里所有人诊断相同”：Mondrian k = 2 时 0.234 的记录处在这样的组里，336 人的真实诊断直接暴露 —— 攻击者不需要知道哪一行是 TA，只要知道 TA 在这个组里。这就是 l-diversity（Machanavajjhala et al. 2006，每组至少 $l$ 种”足够不同”的敏感值）和 t-closeness（Li et al. 2007，组内分布接近总体分布）的动机。我们的诊断有 8 类、分布不算单一，k ≥ 5 后这类泄露就消失了；敏感属性越单一、越和准标识符相关，问题越严重。
3. Background knowledge. k-anonymity 的整个框架假设”攻击者只知道这几列”。下一节说明这个假设在现代数据里是怎么失败的。

2.4 稀疏高维数据：Netflix · High-dimensional sparse data

In 2006 Netflix released 100 million movie ratings from about 500,000 subscribers for the Netflix Prize, with names replaced by random IDs and some ratings perturbed. Narayanan & Shmatikov (2008) showed that this was not anonymous. Their key observation is about sparsity: each subscriber has rated a tiny fraction of the catalog, and in a high-dimensional sparse space no two records are close — your record’s nearest neighbor is still far from you. So a handful of approximate facts about a person (a few movies, roughly when they were rated) is enough to isolate their record. The paper reports that 8 ratings (of which 2 may be completely wrong) with dates known to within ±14 days uniquely identify 99% of records, and 2 ratings with dates within ±3 days identify 68%. Using public IMDb reviews as auxiliary information, they identified Netflix records of known users and the political and other sensitive preferences those records implied. (The second Netflix Prize was cancelled in 2010 after a privacy lawsuit.)

Their Scoreboard algorithm is worth knowing because each design choice teaches something:

• sim tolerates noise: a match counts if the rating is within ±1 and the date within a window — the adversary’s knowledge is approximate.
• wt up-weights rare items: “watched a film that only 30 people rated” says far more than “watched Titanic”. The Spring 2022 slide (summarizing Datta, Sharma & Sinha 2012, who analyzed why the attack works) puts it as “rare attributes become distinguishing attributes because many records have null value”.
• Eccentricity: declare a match only if the best score beats the second best by a margin (here 1.5 standard deviations of all scores); otherwise abstain. This keeps the attack from confidently naming the wrong person — or anyone at all when the target isn’t in the data.

演示 4 的评分库：1 万用户 × 2000 部电影，共 366,363 条评分（每人平均 37 条，填充率 1.83%）；电影热度长尾，最热门的被 9865 人评过、中位数 62 人。攻击者知道目标的 k 条评分（评分 30% 概率差 ±1，日期差 ±14 天以内），300 个目标的正确识别率：

所有设置下认错的比例都是 0.00（eccentricity 检验让攻击不确定时弃权）。稀有度加权在 k 较大时明显更好（k = 6：0.83 vs 0.68），k ≤ 3 时差不多（几条随机评分里本来就未必有冷门电影）。日期是很强的信息；只知道热门电影时攻击更难，但 3 条就够了。

The lesson for k-anonymity: in sparse high-dimensional data there is no fixed set of quasi-identifiers — every attribute can be one. The same pattern has been found in search logs (AOL, 2006: a user was identified by a newspaper from her search queries), location traces, credit-card transactions, and browsing histories. Anonymity is not a property of a dataset alone; it is a property of the dataset together with whatever auxiliary information an adversary has or will have. This is exactly the dependence that differential privacy (L10) is designed to remove.

3. Model inversion 与属性推断 · Model inversion and attribute inference

3.1 华法林：一个端到端的案例

Fredrikson et al.（2014）研究的是一个真实的临床问题。华法林（warfarin）是常用的抗凝药，合适的剂量因人而异、而且剂量错误可能致命；国际华法林药物遗传学联盟（IWPC）的研究表明，加入两个基因型（VKORC1、CYP2C9）能改进剂量预测。作者用 IWPC 数据（训练队列 2644 人、验证队列 853 人）训练了一个预测剂量的线性模型，然后问：模型本身会不会泄露病人的基因型？

攻击者的设定：拿到模型（黑盒即可）、各属性的边际分布、目标病人的人口学信息、以及 TA 实际的稳定剂量。攻击（他们称为 model inversion）是在这些信息下最小化猜错概率的 Bayes 决策：对每个可能的基因型 $v$，计算”先验 × 模型说的剂量和真实剂量吻合的程度”，取最大的那个：

结果（论文原文）：VKORC1 的预测准确率最高到 58%，而只用边际分布猜是 36%；这只比”直接训练一个预测基因型的模型”差约 5%。攻击在训练队列上比在验证队列上准约 3%（AUC 高约 4%），100 次交叉验证中这个差距稳定存在。他们还做了第一个端到端的 DP 评估：ε ≤ 1 的 DP 线性回归能挡住攻击（训练 / 验证队列不再有差别），但在模拟的临床试验里，这么小的 ε 会让病人面临更高的中风、出血和死亡风险。

Fredrikson, Jha & Ristenpart（2015）把思路推广到返回置信度的模型：对人脸识别 API，用置信度做梯度下降，能从一个人名重建出可辨认的人脸图像；对决策树上的生活方式调查，能推断”是否有过外遇”一类的回答。他们也提出了最简单的缓解：只返回粗略的置信度。

3.2 代码 02：反推一个特征

我们在信贷世界里把”近两年逾期次数” missed 当作敏感属性（征信报告里的负面记录），攻击者知道其余 4 个特征。

演示 1：知道模型给 TA 的分数。 把 missed 的每个候选值代进模型，找出能复现这个分数的那个：

这个”攻击”对成员和非成员一样有效 —— 它和训练数据毫无关系。模型是一个确定的函数，知道输出和其余输入，未知的那一个输入就被解出来了。它泄露的是部署时的输入：谁能看到你的分数，谁就知道了你的一部分资料。缓解措施在 API 设计（少返回、返回粗略值），而不在训练算法 —— 差分隐私保护训练数据，对此无能为力。

3.3 是推断，还是泄露？ The member gap

演示 2 做 Fredrikson 式的攻击：攻击者知道 TA 是否还款（$y$），用模型当似然，选 $\arg\max_v P(v)\,P_f(y \mid x_{-s}, v)$。关键是三组对照：目标模型的成员、同分布的非成员、以及一个用另一份数据训练的参考模型（20 次重复取平均；只用先验猜 missed = 0 的准确率是 0.450）：

读这张表：

• 泛化好的线性模型只把准确率从 0.450 提高到 0.468，而且成员、非成员、参考模型完全一样。它学到的是”逾期多的人更容易违约”这条人口规律；用它推断谁的 missed 都一样准。按 §1.4 的区分，这是统计推断。
• 过拟合的三次模型在成员上 0.555、非成员上 0.416（比只用先验还差）。这 0.139 的差距才是训练数据特有的泄露：模型记住了每个成员的（特征，结果）组合，“反过来读”就读出了 missed。
• gap 随过拟合增长（0.000 → 0.014 → 0.073 → 0.139），和成员推断的 advantage 一样。Yeom et al.（2018）把这两者形式化为同一个框架下的 membership advantage 和 attribute advantage，并分析了两者之间的归约关系。

This resolves much of the debate the warfarin paper triggered. Critics (e.g., Frank McSherry’s 2016 essay “Statistical inference considered harmful”) argued that inferring genotype from dose is just what a correct dosing model implies about the population — it would work equally well on patients who never contributed data, so it is not a privacy violation of the training set. Both sides are partly right, and the member/non-member comparison separates them: in the warfarin data, most of the attack’s power was population inference, and a few percentage points on the training cohort were genuine leakage. Whether population inference is acceptable (e.g., an insurer inferring genotype from a public model) is an important ethical and legal question — but it is not one that differential privacy, or any training-time defense, can answer.

4. 成员推断：一个假设检验 · Membership inference as hypothesis testing

4.1 The game

The CS329T membership-inference homework (HW4) defines it precisely. Let the data be split into $T_0$ (non-members) and $T_1$ (the training set of $f$). Flip a fair coin $b \in \{0, 1\}$, draw $x$ uniformly from $T_b$, and give it to the adversary $\mathcal{A}$, who outputs a guess. The adversary’s success is

The HW4 adversary has black-box query access to $f$‘s probability vectors, a shadow dataset $S$ disjoint from the training set, and knowledge of the model type and training algorithm.

为什么”在不在训练集里”是隐私问题？2022 年课上的三个讨论题是很好的出发点：Why is this a significant privacy breach? Who suffers when this happens? Who is held responsible? 如果训练集是”某医院某种病的患者”，成员身份本身就是诊断。历史上第一个引起政策变化的成员推断来自基因组学：Homer et al.（2008）证明可以从全基因组关联研究（GWAS）公布的汇总等位基因频率判断某个人是否在病例组里，美国 NIH 随后把这类汇总数据从公开访问中撤下。成员推断也是整个领域的”度量衡”：审计差分隐私（第 10 讲）、验证 machine unlearning（§8.3）、从语言模型提取训练数据（§9.2）都以它为基本工具。

4.2 最优检验：似然比 · Neyman–Pearson

成员推断是一个假设检验：

攻击者观察到某个统计量 $o$（例如目标模型对 $z$ 的置信度）。TPR 是检验的 power，FPR 是第一类错误率。Neyman–Pearson lemma: for any fixed FPR $\alpha$, the test with the highest TPR rejects $H_0$ when the likelihood ratio

exceeds a threshold chosen so that the FPR equals $\alpha$. Two consequences shape the rest of this lecture:

1. The optimal attack is a likelihood ratio, and the distributions $p(o \mid H_1)$, $p(o \mid H_0)$ are over the randomness of training — which other records were in $S$, the random seed. They are specific to the record $z$: an easy record has low loss under both hypotheses, a hard one high loss under both. A single global threshold on the loss ignores this; per-example tests (§5.3, §5.5) use it.
2. “The best attack” depends on the FPR you operate at. So does the comparison between attacks — which is why a single summary number can mislead.

4.3 为什么要看”低 FPR 下的 TPR” · Why TPR at low FPR

第 1 讲 §5.4 留下了一个悬念：过拟合的三次模型上，loss 攻击的 advantage 有 0.31，AUC 却只有 0.60。代码 03 演示 1 用 256 个模型、102,400 次评估把这件事看清楚（目标模型：训练 200 人；线性模型 train / test acc 0.791 / 0.753，三次模型 0.879 / 0.707）：

按 AUC 和 balanced accuracy，loss 攻击在三次模型上明显高于噪声地板 —— 它”测到了泄露”。但在 FPR = 0.1% 处，它的 TPR（0.11%）和随机分数分不开。另一种看法是 precision：把攻击分数最高的前 k% 评估样本挑出来，里面有多少真是成员？

loss 攻击最有把握的那些判断，和抛硬币差不多：loss 最低的往往是”谁训练都会分对”的容易记录。LiRA（§5.5）最有把握的 0.1% 里，99% 真是成员。

Carlini et al. (2022) made this the central methodological point of “Membership Inference Attacks From First Principles”: privacy is a worst-case notion. An attack that confidently identifies even a small number of training members is a real breach for those people, even if it is no better than chance on everyone else; an attack that is 60% accurate uniformly lets no one be identified with confidence. Average-case metrics (accuracy, AUC) conflate the two, so they recommend:

• report TPR at fixed low FPRs (e.g., 1%, 0.1%, 0.001%) and plot the ROC curve on log–log axes;
• make sure the evaluation has enough non-members to resolve the FPR: at FPR = 0.1% you need thousands of non-members just to have a few false positives — our 51,200 non-members give about 51;
• compare against the floor: a no-signal attack has TPR ≈ FPR, and with finite samples the measured [email protected]% fluctuates (our 95th percentile: 0.13%).

Leino & Fredrikson (2020) had argued the same from a different angle: an attack should be able to make confident positive inferences — high precision — and most earlier attacks could not.

4.4 实验台 · The test bench

代码 03、04 用的是 LiRA 论文的实验设计（privacy_tools.py）：

1. 一个 400 人的候选池；在上面训练 $M = 256$ 个模型，每个随机选 200 人。记录 $M \times 400$ 的成员矩阵 IN 和每个模型对每条记录的 $\phi$。
2. 轮流把每个模型当作目标，其余 255 个当作攻击者的影子模型（shadow models）。每条记录在约一半的影子模型里是成员 —— online LiRA 需要的 IN / OUT 样本都有了。
3. 所有（目标模型, 记录）对拼成一条 ROC：102,400 次评估。

两个实现细节：特征标准化用一份固定的”公开参考样本”的均值和标准差（而不是每个模型自己的训练集），这样所有模型共享设计矩阵、只差参数 $\theta$；它让实验更快，也让”模型之间的差别只来自训练集”。噪声地板用”随机分数 + 同样的成员 / 非成员个数”重复 50 次得到。

5. 攻击 · The attacks

5.1 Gap attack 与 loss 阈值（Yeom et al. 2018）

最简单的两个攻击都只看目标模型自己：

• Gap attack（HW4 的 naive baseline）：分对了就猜”成员”，分错了就猜”非成员”。它的 TPR = 训练准确率、FPR = 测试准确率，所以 advantage = 训练准确率 − 测试准确率：泛化差距本身。
• Loss threshold：$\ell(\theta, z) < \tau$ ⇒ 成员。Yeom et al. 证明，在他们的模型下，这个攻击的 advantage 由泛化误差决定；过拟合是成员推断成功的充分条件 —— 但不是必要条件（§7 会看到，泛化很好的模型上仍有可被认出的人）。

它们是”全局阈值”攻击：对所有记录用同一把尺子。

5.2 Shokri 的影子模型 · Shadow models

Shokri et al. (2017) introduced membership inference against ML models, attacking models trained by commercial ML-as-a-service platforms (Google Prediction API, Amazon ML) and, among other datasets, a hospital-discharge dataset where membership is itself sensitive. Their attack, which HW4 asks you to implement:

1. The attacker has data $S$ from the same distribution (disjoint from the target’s training set). When no such data exist, they synthesize it by querying the target model and keeping inputs it classifies with high confidence.
2. Repeatedly split $S$ into $S_\text{in}$, $S_\text{out}$ and train a shadow model $g$ on $S_\text{in}$ with the same algorithm as the target.
3. Build an attack dataset of $(y, g(x), \text{in/out})$ triples from $g$‘s outputs on $S_\text{in}$ and $S_\text{out}$.
4. Train one attack model per class $m_y$ to predict in/out from the output vector (HW4 uses a small MLP on the probability vector plus a one-hot label).
5. Attack the target by feeding its outputs to $m_y$ — betting that the target behaves on its members the way shadow models behave on theirs.

为什么能成功？影子模型模拟了”同一个训练算法在成员和非成员上的行为差异”，攻击模型学到的是这个差异的形状。后续工作大幅简化了它：Salem et al.（2019，ML-Leaks）发现一个影子模型就够了，影子数据甚至可以来自不同的分布，而且对最大置信度设一个阈值几乎一样好 —— 也就是说，影子模型学到的主要是一个（按类别的）置信度阈值。

我们的实现（ShokriAttack）：攻击者有另外 400 人的数据、16 个影子模型；二分类的输出向量只有一个自由度，所以每个类别的攻击模型是 $\phi$ 的三次多项式逻辑回归。结果见 §5.6 的总表：AUC 0.605，比 loss 阈值（0.571）略好，但低 FPR 处一样贴着地板（[email protected]% = 0.11%）。

5.3 Bayes 最优的视角 · Sablayrolles et al. (2019)

The fireside-chat paper asks: what is the best possible attack, and does white-box access (seeing the parameters) help? They assume the trained parameters follow a posterior of the form

where $m_i \in \{0, 1\}$ indicates membership and $T$ is a “temperature” (a stylized model of SGD’s randomness). Under this assumption they derive that the posterior probability of membership of $z_1$ has, roughly, the form

（论文里是一个对其余成员的期望；这里写成它的近似形式。$\lambda$ 是成员的先验比例。）where $\tau(z_1)$ is a threshold that depends on the sample — essentially the loss that $z_1$ would typically get from models trained without it. Two conclusions:

1. The optimal attack depends on $\theta$ only through the loss $\ell(\theta, z_1)$. Under these assumptions, white-box access gives no advantage over black-box access to the loss — “black-box attacks are as good as white-box attacks”.
2. The threshold should depend on the sample. A global threshold (their MALT) is a coarse approximation; estimating $\tau(z_1)$ per sample with shadow models (their MAST) is better. Existing attacks are coarser approximations of the optimal one.

In our toy, a simple white-box statistic — the per-example gradient norm $\lVert\nabla_\theta\ell\rVert = \sigma(-\phi)\,\lVert a\rVert$ — gives AUC 0.557 vs 0.571 for the loss, and is at the floor at low FPR: no gain from white-box information. The big gain comes from conclusion 2: calibrating per record. Difficulty calibration (Watson et al. 2022) implements it in the simplest way, score $= -(\ell(\theta, z) - \overline{\ell}_\text{OUT}(z))$, using only OUT shadow models: on the cubic model it lifts [email protected]% from 0.11% to 1.25%.

5.4 白盒、校准的攻击 · Leino & Fredrikson (2020)

“Stolen Memories” 从另一个角度研究白盒攻击。他们的洞察是：深度网络过拟合的一个表现是以特异的方式使用特征（idiosyncratic feature use）—— 某些内部特征在训练集里和标签的关系，与在总体里不同；模型利用了这种只在训练集里成立的关系。他们先对一个简化的线性 / 高斯模型推导出 Bayes 最优的白盒攻击，再把它推广到深度网络的各层表示上。主要发现（论文原文的结论）：即使模型的黑盒行为看起来泛化得很好，白盒攻击仍能找到成员身份的证据，并优于之前的黑盒方法；攻击可以被校准到高 precision，而之前的攻击往往不能给出”有把握的阳性判断”；更小的泛化误差不足以防住攻击；小 ε 的差分隐私能降低攻击效果但代价是准确率，而实践中有时使用的较大 ε（例如 ε = 16）下，攻击几乎和对未保护模型一样有效。

这和 Sablayrolles et al. 的”白盒没有帮助”并不矛盾：后者的结论依赖于它对参数后验的假设；在真实的深度网络上，内部表示可能携带 loss 之外的信息。两篇论文共同的主题是 calibration：要么给每条记录自己的阈值，要么让攻击的阳性判断真正可信。

5.5 LiRA：逐条记录的似然比 · The Likelihood Ratio Attack

LiRA (Carlini, Chien, Nasr, Song, Terzis & Tramèr 2022) is the direct implementation of §4.2:

1. For the target record $z$, train shadow models on random subsets of the attacker’s data, about half of which include $z$ (IN) and half not (OUT).
2. Compute the logit-scaled confidence $\phi = \log\frac{p_y}{1-p_y}$ of each shadow model on $z$. The logit transform makes the distribution across training runs approximately Gaussian (raw confidences pile up near 1).
3. Fit $\mathcal{N}(\mu_\text{in}, \sigma_\text{in}^2)$ and $\mathcal{N}(\mu_\text{out}, \sigma_\text{out}^2)$.
4. Online: score $= \log\mathcal{N}(\phi_\text{target}; \mu_\text{in}, \sigma_\text{in}^2) - \log\mathcal{N}(\phi_\text{target}; \mu_\text{out}, \sigma_\text{out}^2)$. Offline: use only the OUT Gaussian in a one-sided test, score $= (\phi_\text{target} - \mu_\text{out})/\sigma_\text{out}$ — the shadow models can be trained once, before the targets are known.
5. With few shadow models, use a global variance shared by all records; query augmentations (e.g., flipped images) add more observations per record.

The paper reports that LiRA is “10× more powerful at low false-positive rates” than prior attacks and also dominates them on average-case metrics.

代码 03 演示 3（对数尺度的 ROC，三次模型）把”差别全在左下角”量化了：

在 FPR = 30% 处，online LiRA 是 loss 攻击的 1.6 倍；在 0.1% 处是 21.4 倍。还有一个值得注意的细节：loss 攻击在 FPR ≤ 30% 的每一列都紧贴随机线 —— 它的 balanced accuracy 0.584 来自 FPR 很高的区域：loss 很高的记录几乎肯定不是成员。loss 攻击擅长”排除”，不擅长”指认”。

演示 4（消融，[email protected]% FPR）：

• 影子模型越多，online LiRA 越强；少于约 32 个时逐条估计的方差太吵，全局方差更好 —— 和论文的建议一致。影子模型少时，最简单的 calibrated loss 是最强的攻击。
• 统计量的选择很关键：同样的 online 高斯拟合，用 $\phi$ 时 2.42%，用概率 $p_y$ 时 1.35%，用 $\log p_y$（= −loss）时只有 0.07%。高斯假设只在统计量真的接近高斯时成立。

一个和论文不同的结果：offline LiRA。 论文在 CIFAR-10 上报告 offline 只比 online 弱一些；在我们的三次模型上，offline LiRA 在任何影子模型数量下都停在地板上（0.08–0.14%），而同样只用 OUT 模型的 calibrated loss 有 1.25%。演示 4 的诊断表看的是 OUT 分布（$\phi$ 在不含该记录的模型上的分布）的形状：

被误报的记录 OUT 分布明显右偏：偶尔会在某个 OUT 模型上得到很高的 $\phi$，高斯单边检验把这种右尾的偶然值当成了”3σ 的罕见事件”；而最常被记住的记录 $\sigma_\text{out}$ 最大，除以 $\sigma_\text{out}$ 又把它们的信号压小了。也就是说，高斯假设在尾部失效，而低 FPR 恰恰只看尾部。教训：哪种统计量最好取决于模型和数据，评估隐私时应该同时跑几种攻击，报告最强的那个。

演示 5 用两条记录说明”逐条阈值”在做什么（目标模型 #0）：

全局阈值几乎第一个就把 #130 当成成员，但它在不在训练集里 loss 都是 0；#271 的 loss 0.354 看起来平平无奇，可它不在训练集里时平均 loss 是 9.66 —— 相对于它自己的基线，0.354 低得反常。这就是 Sablayrolles et al. 的 $\tau(z)$，LiRA 用影子模型把它估计出来。

5.6 一张总表 · The attack ladder

代码 03 演示 2，所有攻击在同样的 102,400 次评估上：

Two things to notice beyond the ranking. First, on the well-generalizing linear model every attack has AUC ≤ 0.556 — by average-case standards it “doesn’t leak” — yet calibrated loss reaches 0.43% at FPR 0.1%, about three times the floor’s 95th percentile: a few records can still be identified (§7 shows who). Second, the strongest attacks need more knowledge (the candidate pool and many shadow models). That is the right assumption for an audit — you want to know what a strong adversary could do — and it is also what makes these attacks expensive. Newer attacks reduce the cost: RMIA (Zarifzadeh et al. 2024) compares the target’s likelihood ratio against population samples and reports strong low-FPR performance with few reference models; Ye et al. (2022) systematize the “reference model” family of attacks.

6. 怎样报告隐私风险 · A reporting checklist

把前两节的教训压缩成一张清单（适用于你自己的模型、论文审稿、或一份隐私评估报告）：

1. Threat model：攻击者能看到什么（标签 / 置信度 / 参数）、有什么数据、知道训练算法吗？审计时应假设强攻击者。
2. Strongest attack：至少跑 loss 阈值、校准的 loss 和一个 LiRA 类攻击；不同模型上最强的攻击可能不同（我们的 offline LiRA 就是例子）。
3. TPR at fixed low FPRs + log–log ROC；AUC 和 balanced accuracy 作为补充，不作为主要结论。
4. Noise floor and sample size：非成员的数量决定了能测量的最低 FPR；报告零信号攻击在同样样本量下的值（均值和 95% 分位数）。
5. Per-record risk：报告最危险的记录是谁、属于哪些子群体（§7），而不只是一个平均数。
6. “No attack succeeded” ≠ “private”：一个攻击失败只说明这个攻击失败。可证明的上界只能来自差分隐私（第 10 讲），而经验攻击可以给出 DP 参数 ε 的下界（privacy auditing：Jagielski, Ullman & Oprea 2020；Nasr et al. 2021；Steinke, Nasr & Jagielski 2023）。

7. 谁会被认出来 · Who is vulnerable

7.1 逐条记录的风险 · Per-record risk

平均泄露很小，不代表每个人都安全。代码 04 演示 1 在候选池里埋了两类特殊记录（各 8 条）：特征离群（收入 ×3、逾期次数 +4，两个互相矛盾的强信号，标签按真实概率抽）和标签反常（标签和真实还款概率相反，对应录入错误或真正的例外）。对每条记录，用 256 个模型里 IN / OUT 两组 $\phi$ 计算 per-record KS advantage $\max_\tau[P_\text{in}(\phi > \tau) - P_\text{out}(\phi > \tau)]$ —— “专门针对这个人、并且知道两种世界下模型输出分布”的最好攻击的 TPR − FPR —— 以及一个按 1% 全局 FPR 工作的 online LiRA 认出 TA 的比例：

（KS 的噪声地板：打乱 IN 标签后平均 0.074，95% 分位数 0.142。）

• 离群者最危险：在训练集里时 loss 中位数 0.047，不在时 11.5。模型在”没有同伴”的区域外推出极端的预测，只有亲眼见过 TA 才会把 TA 分对。
• 标签反常者：模型必须”弯折”才能拟合 TA 的标签（在 / 不在时 0.787 / 1.579）。
• 少数群体：group 1 的普通记录比 group 0 更容易泄露（0.301 vs 0.246），因为 thin-file 效应让 TA 们的标签更难从特征预测，模型只能更多地”记”。但这个差距远小于”反常”带来的差距：在这个世界里，反常比少数更危险；而少数群体里”反常”的人更多。
• 风险高度集中：普通记录里最危险的 10% 平均 KS 0.729，最安全的一半 0.124；16 条特殊记录（4% 的人）贡献了 1% FPR 下全部命中的 16.4%。
• 线性模型把一切压平了（普通记录 0.129，地板 0.075），但特殊记录被认出的比例（0.100 / 0.091）仍是普通人（0.015）的 6 倍以上 —— 这就是 §5.6 里线性模型”整体不泄露、低 FPR 处仍有信号”的来源。

Long et al. (2018) reported the same pattern on real data: even well-generalized models leak membership of a small set of vulnerable, atypical records. Kulynych et al. (2022) call the group-level version “disparate vulnerability” — and it is the privacy counterpart of L1 §5.2: averages hide individuals and subgroups.

7.2 重复的数据 · Duplicates

演示 1b 是一个配对比较：同样的 40 个普通人，在训练集里出现 1、2、4、8 次（其他一切相同）：

这 40 个人的特征和标签完全正常，只是数据管道里出现了重复。重复 = 训练目标里更大的权重 = 更深的记忆；连线性模型都被它”教会”了记忆。其他人反而略微更安全了（模型的”记忆容量”被重复的人占用）。这是语言模型里”重复越多、越容易被逐字提取”（§9）的表格版。

7.3 记忆与泛化 · Memorization vs generalization

为什么最好的模型偏偏要记住这些人？Feldman（2020）给出了一个理论回答（“Does learning require memorization? A short tale about a long tail”）：现实数据的子群体频率是长尾的，很多子群体在训练集里只出现一两次；要在这些罕见子群体的测试样本上表现好，算法就必须记住它们的训练样本 —— 而且这种记忆在长尾分布下对泛化误差是必要的，不是过拟合的副作用。Feldman & Zhang（2020）在图像分类上估计了每个样本的 memorization score（包含 / 不包含该样本时模型答对它的概率之差），并发现去掉高记忆分数的样本会明显降低测试准确率。

这和第 4 讲 §10.4 的链条是同一件事 —— 罕见、离群、标签出人意料 → 模型必须为 TA 专门弯折 → 自影响高 → 在 / 不在训练集里差别大 → 成员身份泄露 —— 那里用影响函数排序，这里用 LiRA 测量。它也解释了 Fall 2023 和 Autumn 2025 第一讲那张幻灯片（见第 1 讲 §6.3）：几十年的 ML 研究优化的是泛化、惩罚的是记忆，但两者的重叠 “is really murky”。对隐私而言，这意味着一个令人不安的结论：完全不记忆、又在长尾上表现好，可能根本做不到；差分隐私对少数群体的准确率代价（第 1 讲 §7.4，Bagdasaryan et al. 2019）正是这个张力的另一面。

7.4 隐私洋葱 · The privacy onion

如果风险集中在少数人身上，把这些人删掉不就行了？Carlini et al.（2022，“The Privacy Onion Effect: Memorization is Relative”）发现：删掉最脆弱的一层样本、重新训练后，原本安全的下一层会变得脆弱。代码 04 演示 2：删掉 KS 最高的 10%（40 人），在剩下的 360 人上重新做整套实验：

第二层的 KS 升了 0.084（其中 60% 的人上升），是”背景上升”（其余 80%：+0.028，来自训练集从 200 人变成 180 人）的 3 倍；而且第二层是按有噪声的 KS 挑出来的，回归均值本该把它的”之后”拉低。整体攻击能力也没有下降（online LiRA 的 [email protected]%：1.09% → 1.71%；AUC 0.670 → 0.662）。“最危险”是相对的：新的数据里总有人处在最边缘。所以”找出高风险记录然后删掉”不是一种防御。

7.5 正则化与训练集大小 · Regularization and data size

第 1 讲 §7.2 用 loss 攻击得到”隐私和效用先一起变好、后来才需要取舍”。代码 04 演示 3 用 LiRA（每格 128 个模型）重做，结果只对了一半：

• 训练集大小的效果干净而单调：n 从 50 到 800，LiRA 的 [email protected]% 从 4.86% 降到 0.35%，测试准确率从 0.669 升到 0.754。每个人对模型的”份量”是 $1/n$ 量级，更多数据同时带来效用和隐私。
• 正则化不是这样。λ 从 1e-4 到 0.1，泛化差距、loss 攻击的 AUC 都在下降，测试准确率在上升 —— 按第 1 讲的标准模型”更安全了”；但 LiRA 的 [email protected]% 从 0.48% 升到 1.83%。只有 λ ≥ 1（欠拟合）时才明显下降。我原本预期它单调下降。一个解释：λ 很小时，三次模型在 OUT 记录上的输出极不稳定（OUT 分布很宽），成员带来的偏移被淹没在噪声里；适度正则化让 OUT 分布变窄，同样的偏移变得显眼了。泄露是信噪比，正则化同时压低了信号和噪声。
• 教训：泛化差距和 loss-attack AUC 是平均泄露的代理，不能代替”最强攻击在低 FPR 处”的测量。“正则化可以防成员推断”只在正则化强到牺牲准确率时才可靠成立。

8. 不靠 DP 的防御与机器遗忘 · Defenses short of DP, and machine unlearning

8.1 防御一览 · The menu

8.2 只返回标签也挡不住 · Label-only attacks

Choquette-Choo, Tramèr, Carlini & Papernot (2021) showed that hiding confidences doesn’t hide membership: with label-only access, an attacker can query perturbed versions of the target record (data augmentations, or random noise) and use how robustly it is classified — essentially its distance to the decision boundary — as a stand-in for confidence. They found that label-only attacks match confidence-based ones, that confidence-masking defenses (including MemGuard) therefore fail, and that the defenses that held up were differential privacy and strong L2 regularization.

代码 04 演示 4：每条记录查询 64 次 $x + \mathcal{N}(0, (0.3\,\text{sd})^2)$，只看标签对不对：

在 1% FPR 处，标签攻击找回了完整置信度 LiRA 约 80% 的能力，远高于”完整置信度 + loss 阈值”。在最极端的 0.1% 处它弱得多：64 次查询的比例只有 65 个可能值，很多记录”64 个邻居全分对”、并列最高分（所以直接用阈值的那一行在低 FPR 处是 0）。更多查询、更聪明的查询会缩小这个差距。结论：confidence masking 改变的是攻击的成本，不是信息本身。

8.3 Machine unlearning

The GDPR’s right to erasure (Article 17) and similar laws (e.g., California’s CCPA) let people ask organizations to delete their data. Deleting a row from a database is easy; the question raised by this lecture is whether the model still remembers it. Cao & Yang (2015) coined “machine unlearning”; the gold standard is obvious and expensive — retrain from scratch without the deleted data. Everything else trades exactness for cost:

• SISA (Bourtoule et al. 2021: Sharded, Isolated, Sliced, Aggregated): split the training data into $S$ disjoint shards, train one model per shard (optionally checkpointing over slices), and aggregate their predictions. Deleting a point only requires retraining its shard (from the last checkpoint before its slice). Unlearning is exact — the retrained shard never saw the point — at a cost in accuracy that grows with $S$ and with task difficulty.
• Influence / Newton updates: for convex models, one Newton step on the remaining data from the trained parameters approximates retraining (the same approximation as L4’s influence functions); Guo et al. (2020) combine it with noise to obtain certified removal, a DP-style guarantee that the result is indistinguishable from retraining.
• Heuristic approximate unlearning: gradient ascent on the forget set, fine-tuning on the retained data, and many variants — cheap, popular for large models, with no guarantee.

How do you verify that unlearning worked? The natural test is a membership inference attack on the forgotten records: after unlearning they should look like non-members. Two pitfalls: forgotten records can look too forgotten (their loss far above that of genuine non-members — a “Streisand effect” that itself reveals who was deleted), and weak MIAs make inexact unlearning look better than it is — Hayes et al. (2024) show that per-example attacks like LiRA are much stronger evaluators than the population-level attacks commonly used in the unlearning literature. A deeper caveat (Thudi et al. 2022): because different datasets can lead to the same parameters, “this model has unlearned $z$” cannot be verified from the parameters alone; meaningful claims must be about the algorithm that produced them.

代码 04 演示 5：16 个三次目标模型，每个随机遗忘 10 个成员（共 160 条），五种做法，用 255 个影子模型做 online / offline LiRA 检验：

• 重训后的被遗忘者和真正的非成员看不出区别（“过度遗忘”0.031 略高于 0.008：重训模型只有 190 人，和参考模型的设定不完全一样）。一步 Newton 很接近重训。
• 温和的梯度上升通过了逐条的 MIA 检查（0.019，LLR > 0 为 0.463），但它在全部 400 条记录上和重训模型的输出差距（0.721）比不遗忘（0.678）还大：它只是把被删者附近”按平”了，被删者对其他人的影响还留在模型里。逐条的 MIA 检查是必要条件，不是充分条件。
• 强烈的梯度上升把被遗忘者的 loss 推到 35.5、测试准确率掉到 0.473，0.744 的人”过度遗忘”—— 审计者一眼就能看出”这些人被刻意抹掉了”。（online LiRA 反而把其中 0.081 判为成员：观测值远离两个高斯时，似然比由两者的方差决定 —— 又一个高斯假设在尾部失效的例子。）遗忘的目标是”和从没见过 TA 的模型不可区分”，不是”loss 越大越好”。
• SISA：分 2 / 4 / 8 片时，遗忘一条只需重训 50% / 25% / 12.5% 的数据。我预期会损失准确率，结果三次模型 0.681 → 0.714 / 0.696 / 0.685，线性模型 0.747 → 0.750 / 0.755 / 0.753 —— 几乎没有代价：单个三次模型在 200 条上过拟合，平均几个小模型反而起了正则化作用。SISA 论文在更复杂的任务上报告了分片越多准确率下降越明显；代价取决于模型和数据。

9. 语言模型：记忆与训练数据提取 · Memorization and extraction in language models

For generative models, membership inference has a more direct cousin: making the model say its training data. 代码 05 uses a character-level n-gram model (char_lm.py) — the most transparent memorizer there is, whose “capacity” is its order $n$ — trained on 6000 synthetic sentences (172,484 characters) such as “my code is 4-9-8-5-3.” and “trent lost the key on thursday.”. About 287 of them are “my pin is d-d-d-d.” with other people’s random PINs.

The average gap is small (most of the corpus is generalizable templates), but as we will see, small average overfitting hides near-perfect memorization of specific secrets.

9.1 Canaries and exposure（“The Secret Sharer”）

Carlini, Liu, Erlingsson, Kos & Song (2019) proposed a clean way to measure unintended memorization: insert a canary — a synthetic secret such as “my social security number is 078-05-1120” — into the training data, train, and ask how highly the model ranks the true secret among all possible fillings $\mathcal{R}$ of the same template:

Exposure is in bits: each extra bit halves the number of guesses an adversary needs (who enumerates candidates from most to least likely). A secret ranked first has exposure $\log_2|\mathcal{R}|$; a secret the model knows nothing about has an expected exposure of about $\log_2 e \approx 1.44$ bits (练习 4). For large $|\mathcal{R}|$ the paper estimates the rank by sampling and fitting a distribution; with our 4-digit PIN ($|\mathcal{R}| = 10^4$, max 13.29 bits) we enumerate every candidate exactly. The paper found that memorization of canaries is unintended and persistent: it happens early in training, is not prevented by standard regularizers, and was eliminated in their experiments only by differentially private training; they also applied the method to Google’s Smart Compose, a text-completion model trained on millions of users’ emails.

代码 05 演示 2（每格是 5 个 canary 的平均 exposure，bit）：

• Capacity：3 阶模型只看得见两个字符的上下文（“4-”、“-8”），这些上下文在别的号码里出现过成百上千次，canary 要重复 16 次 exposure 才到 9.12。9 阶模型见过一次的 canary 就有 7.00 bit，见过 4 次到上限 —— 按概率猜，第一次就中。
• Duplication：每一行都随重复次数单调上升。去重（r 从 16 变成 1）把 7 阶模型的 exposure 从 13.29 降到 6.25 —— 有效，但不归零。
• Floor：r = 0 一列是 5 个从没插入过的秘密，有噪声（1.42–2.67）；演示 4 用 9717 个从没出现的 PIN 得到 1.33–1.43 bit，和 1.44 一致。

9.2 从 GPT-2 提取训练数据 · Extraction

Carlini et al. (2021, “Extracting Training Data from Large Language Models”) showed that the threat is not hypothetical. Against GPT-2 they (1) generated 600,000 samples (200,000 for each of three generation strategies), (2) ranked them with membership-inference signals — low perplexity, and especially perplexity relative to a reference (the zlib compression entropy of the text, or the perplexity under a smaller model), which filters out text that is merely generic — and (3) manually checked the top 1800 candidates against the web. Several hundred (604 in the paper’s count) were verbatim training data: news, code, log files, and personally identifiable information — names, phone numbers, email addresses — some of which appeared in only a single document in the training data. Larger models memorized more: in one experiment the paper reports that the 1.5B-parameter GPT-2 XL memorized over 18× as much as the 124M-parameter GPT-2 Small. The paper defines k-eidetic memorization: a string is memorized if it can be extracted and appears in at most $k$ training documents — the rarer, the more worrying.

代码 05 演示 3 看”能不能让模型原样说出来”（5 个 canary 中成功的个数；rank 1 / rank ≤ 10 = 让攻击者猜 1 次 / 10 次）：

贪心解码严重低估了记忆：7 阶模型在 r = 4 时 5 个 canary 全部排第 1（exposure 达到上限），贪心解码却一个也说不出来，要到 r = 16 才全部说出来。原因是第一个数字的上下文 “in is ” 后面跟过几百个别人的 PIN：canary 的第一个数字要重复很多次才能成为”最常见的下一个字符”，而后面几步的上下文只属于 canary。束搜索看得更远（r = 4 时 4 / 5）。论文里的提取大多用贪心解码或采样，所以给出的是记忆的下界。

演示 4：语料里本来就有的、别人只出现一次的 PIN（279 个），模型记住了吗？

攻击者照着 9 阶模型的概率猜 100 次，能猜中约六分之一的”语料里真实存在的 PIN”。canary 只是测量工具；真正的风险是语料里本来就有的秘密。

9.3 记忆的规模规律 · Memorization scales

Carlini, Ippolito, Jagielski, Lee, Tramèr & Zhang (2023, “Quantifying Memorization Across Neural Language Models”) defined a sequence as extractable if prompting the model with its $k$-token prefix from the training data makes greedy decoding reproduce the continuation exactly. Across the GPT-Neo family trained on the Pile, they found three log-linear relationships: memorization grows with (1) model capacity, (2) the number of times an example is duplicated in the training data, and (3) the number of tokens of context used to prompt the model. They estimate that the 6-billion-parameter GPT-J memorized at least 1% of its training data under this definition, and concluded that memorization is more prevalent than previously believed and likely to grow with scale absent mitigations.

我们的玩具重现了这三个方向。（1）容量和（2）重复见 §9.1 的表。（3）已知前缀的长度（9 阶模型，r = 2）：

“is ” 后面还跟着 room、code、seat 等各种号码，前缀越短，模型越难把它和 canary 联系起来。（n-gram 只看最后 $n - 1$ 个字符，所以更长的前缀不再有用 —— 前三行几乎相同；Transformer 没有这个硬上限。）

9.4 生产模型与 LLM 上的成员推断 · Production models and MIAs on LLMs

• Aligned chat models still regurgitate. Nasr et al. (2023, “Scalable Extraction of Training Data from (Production) Language Models”) found a “divergence” attack on ChatGPT (gpt-3.5-turbo): asking it to repeat a word such as “poem” forever eventually made it diverge from chat-style answers and emit memorized pretraining text at a much higher rate; with about $200 of queries they extracted over 10,000 unique verbatim training examples. Alignment changes what a model usually says, not what it has memorized.
• Membership inference on pretraining data is hard — and easy to mis-measure. Pretraining typically sees each document about once, on trillions of tokens, and the boundary between “member” and “non-member” text is fuzzy (near-duplicates, quotations). Duan et al. (2024, “Do Membership Inference Attacks Work on Large Language Models?”) found that existing MIAs barely beat random guessing on LLM pretraining data in most settings, and that apparent successes in some benchmarks came from a distribution shift between members and non-members (e.g., non-members collected after the training cutoff differ in topic and time) — a temporal version of L1’s “the test set shares the pipeline’s flaws”. Detection methods such as Min-K% Prob (Shi et al. 2024) score text by its least likely tokens; they must be evaluated on carefully matched member / non-member sets.
• Influence functions as a microscope. In the Fall 2023 guest lecture (L4 §10.2), Grosse et al. (2023) found that most behaviors of large models are influenced by many training sequences rather than a few — but for famous quotes, the top influential sequences contained the exact passage. Memorization and generalization coexist in the same model.

9.5 缓解 · Mitigations for LMs

• Deduplication. Lee et al. (2022) showed that removing near-duplicate text from training corpora reduces the rate at which models emit memorized text and also makes training more efficient; Kandpal, Wallace & Raffel (2022) found that the rate at which a sequence is regenerated grows superlinearly with how often it is duplicated, and that deduplication substantially lowers MIA and extraction success. 代码 04 演示 1b 和 05 演示 2 是同一件事的两个玩具版本。
• Filtering PII and secrets before training; output filters that block verbatim matches with known training data (e.g., GitHub Copilot offers a filter that suppresses suggestions matching public code).
• DP training or DP fine-tuning (L10) — the only mitigation with a guarantee for single-occurrence secrets, which deduplication cannot help.

9.6 版权与”反刍” · Copyright and regurgitation

Fall 2024 第 3 讲把 privacy、unlearning 和 copyright 放在了一起，因为它们共享同一个技术问题：模型里存了多少训练数据的原文？几个要点（只讲技术侧，不做法律判断）：

• Memorization is measurable; copying is a legal question. 逐字反刍（regurgitation）是最直接的证据：本讲的 exposure 和提取率都可以用来测量某个作品被记住了多少。2023 年 12 月 The New York Times 起诉 OpenAI 和 Microsoft，诉状的核心证据之一就是模型能逐字复述大段报道。
• The same mitigations reappear：去重（被大量转载的文本被记得最牢）、输出过滤、以及针对特定内容的 unlearning。
• LLM unlearning is hard to evaluate. Eldan & Russinovich (2023, “Who’s Harry Potter?”) fine-tuned a model to “forget” the Harry Potter books; follow-up benchmarks such as TOFU (Maini et al. 2024) evaluate unlearning on fictitious author profiles. §8.3’s lessons apply directly: passing one test (the model no longer completes a passage) is not the same as being indistinguishable from a model never trained on the data.

10. 小结 · Summary

Six takeaways:

1. Anonymity is a property of data plus auxiliary information, not of a dataset alone. Quasi-identifiers are everywhere, and in sparse high-dimensional data every attribute is one.
2. Separate inference from leakage. What a model reveals about anyone (population inference) and what it reveals because you were in the training set (membership, member gap) are different harms with different remedies.
3. Membership inference is a hypothesis test, and the optimal test is a per-record likelihood ratio. That is why calibrated attacks (difficulty calibration, LiRA) beat global thresholds by an order of magnitude at low FPR.
4. Report TPR at low FPR with a noise floor. Average-case metrics hide whether anyone can be identified with confidence.
5. Risk is individual. Outliers, surprising labels, duplicates and rare subgroups carry most of it; removing them exposes the next layer; average generalization gaps are poor proxies.
6. Defenses short of DP raise the cost but give no guarantee — label-only attacks bypass masking, approximate unlearning can pass weak checks, deduplication doesn’t protect single-occurrence secrets. The guarantee comes next lecture.

11. 常见误区 · Common pitfalls

• “We removed names and IDs, so the data are anonymous.” Quasi-identifiers (§2.1) and sparse attributes (§2.4) re-identify people; “anonymous” depends on what the adversary knows.
• “The dataset is k-anonymous, so no one’s diagnosis is exposed.” Homogeneous groups leak the sensitive value without re-identification (§2.3).
• “The model inversion attack works, so the model leaks its training data.” Check it on non-members; if it works equally well, it is population inference (§3.3).
• “Our MIA has AUC 0.57, so the leak is small” — or “AUC 0.6, a serious leak.” Neither follows; look at TPR at FPR 0.1% against a noise floor (§4.3).
• “We measured [email protected]% on 500 non-members.” That is half a false positive; the number is noise (§4.3).
• “No attack beat chance, so the model is private.” A failed attack bounds nothing; use the strongest attacks you can, report per-record risk, and use DP for guarantees (§6).
• “Our train–test gap is tiny, so membership is safe.” Vulnerable records leak even from well-generalizing models, and regularization can even raise low-FPR risk (§7.1, §7.5).
• “We return only labels.” Repeated perturbed queries recover most of the signal (§8.2).
• “We fine-tuned the model to forget them, and the MIA says they’re forgotten.” Check against a retrained model, and beware the Streisand effect (§8.3).
• “Greedy decoding doesn’t produce the secret, so it isn’t memorized.” Exposure and beam search can show full memorization that greedy decoding misses (§9.2).

12. 练习 · Exercises

练习 1（AUC 与低 FPR 的 TPR 可以完全脱节）. Suppose the attack score of every non-member is $\mathcal{N}(0, 1)$. (a) Model A (“uniform leakage”): every member’s score is $\mathcal{N}(1, 1)$. Compute the AUC and the TPR at FPR = 0.1%. (b) Model B (“concentrated leakage”): 98% of members have score $\mathcal{N}(0, 1)$ and 2% have $\mathcal{N}(5, 1)$. Compute the AUC and the TPR at FPR = 0.1%. (c) Which model is “more private”? Relate to §4.3.

(a) AUC $= P(X_\text{in} > X_\text{out}) = \Phi(1/\sqrt2) = \Phi(0.707) \approx 0.760$。阈值 $\Phi^{-1}(0.999) \approx 3.09$，TPR $= \Phi(1 - 3.09) = \Phi(-2.09) \approx 0.018$。(b) AUC $= 0.98 \times 0.5 + 0.02 \times \Phi(5/\sqrt2) \approx 0.49 + 0.02 \times 0.9998 \approx 0.510$；TPR $= 0.98 \times 0.001 + 0.02 \times \Phi(5 - 3.09) \approx 0.001 + 0.02 \times 0.972 \approx 0.020$。(c) 两者在 FPR = 0.1% 处能确定地认出的成员比例几乎一样（约 2%），AUC 却是 0.76 vs 0.51。对 B 里那 2% 的人来说，B 远不私密：TA 们几乎一定会被认出（TPR 0.97）；而在 A 里，每个人被认出的概率都只有约 2%。只看 AUC 会说 B 几乎不泄露 —— 这正是代码 03 里线性模型的样子（AUC ≤ 0.556，低 FPR 处仍有信号）。

练习 2（Gap attack 与泛化差距）. (a) Show that the gap attack (“member iff correctly classified”) has TPR = training accuracy and FPR = test accuracy (on non-members from the same distribution), so its advantage equals the generalization gap. (b) In code 03, the cubic model has train / test accuracy 0.879 / 0.707 and the gap attack’s balanced accuracy is 0.583. Is this consistent? (c) Why can this attack never have nonzero TPR at FPR = 0.1%?

(a) 攻击把”分对”判为成员：成员里被判为成员的比例 = 成员中分对的比例 = 训练准确率；非成员里被判为成员的比例 = 非成员中分对的比例 = 测试准确率。advantage = TPR − FPR = 泛化差距（Yeom et al. 2018）。(b) $0.5 + (0.879 - 0.707)/2 = 0.586$；表中是 0.583，因为非成员是候选池里的非成员（每个目标模型的另外 200 人），而不是 20000 人的大测试集，测试准确率略有不同。(c) 分数只有两个值。阈值要么”全判非成员”（TPR = FPR = 0），要么”分对的都判成员”（FPR ≈ 0.707）。FPR ≤ 0.1% 只能取前者。

练习 3（准标识符的唯一性）. A ZIP code has $m$ residents, and each person’s (gender, date of birth) falls uniformly into one of $D = 2 \times 73 \times 365$ cells. (a) Show that a given person is unique with probability $(1 - 1/D)^{m-1} \approx e^{-(m-1)/D}$. (b) Evaluate it for $m$ = 2000, 8000 and 30000. (c) Compare with the by-ZIP-size table in §2.1 and explain the remaining differences.

(a) 其余 $m - 1$ 人每人以 $1 - 1/D$ 的概率不和 TA 同格，相互独立。(b) $D = 53{,}290$：$e^{-1999/53290} \approx 0.963$，$e^{-7999/53290} \approx 0.861$，$e^{-29999/53290} \approx 0.570$。(c) 表中是 0.952（< 4000 人）、0.865（4000–10000）、0.725（10000–25000）、0.548（≥ 25000），量级吻合。差别来自：每组里邮编的大小不一（取的是组内平均）；年龄分布不均匀（年轻人多），出生日期并不是均匀地落在 $D$ 个格子里 —— 格子不均匀时碰撞更多，唯一的比例更低。真实人口里出生日期也不均匀（某些月份出生更多），邮编大小差别更大。（Golle 的 63% 和 Sweeney 的 87% 之间的差别，还涉及两次普查的数据和估计方法的不同。）

练习 4（exposure 的噪声地板）. (a) If the model knows nothing about a secret, its rank among $|\mathcal{R}|$ candidates is uniform on $\{1, \dots, |\mathcal{R}|\}$. Show that $\mathbb{E}[\text{exposure}] \approx \log_2 e \approx 1.44$ bits for large $|\mathcal{R}|$. (b) An exposure of 7 bits with $|\mathcal{R}| = 10^4$ corresponds to which rank? (c) Why is the r = 0 column of §9.1 (1.42–2.67) noisier than demo 4’s 1.33–1.43?

(a) $\text{exposure} = \log_2(|\mathcal{R}|/\text{rank})$。令 $u = \text{rank}/|\mathcal{R}|$ 近似均匀分布在 (0, 1]，$\mathbb{E}[-\log_2 u] = \int_0^1 -\log_2 u\,du = 1/\ln 2 \approx 1.443$。(b) $\text{rank} = 10^4/2^7 \approx 78$：排在第 78 位左右，前 0.8%。(c) r = 0 一列每格只是 5 个秘密的平均，单个秘密的 exposure 的标准差约为 $1/\ln 2 \approx 1.44$ bit（指数分布），5 个平均后标准误约 0.65 bit；演示 4 平均了 9717 个从没出现的 PIN。另外 n-gram 模型对没见过的候选常常给出完全相同的概率（并列），我们的排名规则（严格更高的个数 + 1）在并列时对攻击者有利，会让地板略微偏高。

练习 5（遗忘的验证）. In §8.3, mild gradient ascent passed the per-record MIA check but remained far from the retrained model. (a) Propose a test that would catch it, using only things an auditor could compute. (b) Why does the “too forgotten” rate of the strong gradient-ascent model reveal who was deleted? (c) Thudi et al. (2022) argue that unlearning must be defined for algorithms, not for models. Give an intuitive reason based on L4’s leave-one-out.

(a) 和”从头重训”的模型（或一组用同样数据、不同种子重训的模型）比较所有记录的输出分布，而不只是被删者的：例如被删者的近邻、同一子群体的记录上，遗忘后模型的输出是否落在重训模型的分布内（一个分布层面的两样本检验）；也可以对被删者用更强的逐条攻击（LiRA，Hayes et al. 2024）。(b) 被遗忘者的 loss 远高于真正的非成员（中位数 35.5），offline z 分数远低于 −2.33。审计者只要找出”模型对 TA 的置信度比任何没见过 TA 的模型都低得多”的人，就找到了被删除的人 —— 删除请求本身成了新的泄露（Streisand effect）。(c) 第 4 讲 §2–§4：参数是许多训练样本共同决定的，删掉一个样本对参数的影响可能小于随机种子带来的差异；不同的训练集（甚至不包含 $z$ 的训练集）也可能得到几乎相同的参数。所以只看最终参数，无法判断它”是否用 $z$ 训练过”；可验证的说法只能是”这个算法的输出分布和不包含 $z$ 时的分布不可区分”—— 这正是第 10 讲差分隐私的语言。

练习 6（为一个真实部署写威胁模型）. A hospital wants to expose a 30-day readmission risk model as an API to partner clinics. For each attack in this lecture (linkage, attribute inference, membership inference, extraction if applicable), state (a) what the attacker would need, (b) what you would measure and how (metric, sample size, noise floor), (c) which records you expect to be most at risk, and (d) one mitigation and its limitation.

• Membership inference（最相关：成员身份 = “在这家医院住过院”）。(a) 返回概率的 API + 同分布数据（合作诊所可能有）。(b) 用 LiRA（影子模型 ≥ 64 个，或少时用 calibrated loss）在候选池上测 [email protected]% FPR，非成员至少数万次评估，报告随机分数的地板。(c) 罕见诊断组合、极端化验值、重复入院（同一个人多条记录 —— §7.2 的重复）、标签反常的病人。(d) 只返回风险等级（三档）：提高攻击成本，但 label-only 攻击仍能部分恢复（§8.2）；真正的保证要用 DP 训练（第 10 讲），并评估它对罕见病人准确率的影响。
• Attribute inference。(a) 知道病人的部分特征 + 风险分数。(b) 在成员和非成员上分别测推断准确率，报告 member gap（§3.3）。(c) 与敏感属性（如精神疾病诊断）强相关的特征组合。(d) 返回粗略分数；但对非成员同样有效的”统计推断”部分，只能靠限制谁能查询、查询什么来管理。
• Linkage：如果同时发布任何训练数据的汇总或样本，检查准标识符的唯一性（§2.1）；不发布记录级数据。
• Extraction：表格模型不适用；如果系统里有生成式组件（比如自动生成出院小结的 LLM），需要 canary 测试（§9.1）和去重。

代码相关的练习（“动手改一改”）见 code/lecture09/README.md。

13. 延伸阅读 · Further reading

本讲对应的原课材料

• CS329T Spring 2022 “Privacy, Week 1 (only week)” slides（Datta & Mitchell）与 HW4（Shokri 式影子模型攻击）。
• Narayanan & Shmatikov, “Robust De-anonymization of Large Sparse Datasets”, IEEE S&P 2008.
• Fredrikson, Lantz, Jha, Lin, Page & Ristenpart, “Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing”, USENIX Security 2014.
• Sablayrolles, Douze, Ollivier, Schmid & Jégou, “White-box vs Black-box: Bayes Optimal Strategies for Membership Inference”, ICML 2019.
• Shokri, Stronati, Song & Shmatikov, “Membership Inference Attacks Against Machine Learning Models”, IEEE S&P 2017.
• Leino & Fredrikson, “Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference”, USENIX Security 2020.

Re-identification

• Sweeney, “Simple Demographics Often Identify People Uniquely”, CMU 2000；Sweeney, “k-Anonymity: A Model for Protecting Privacy”, 2002；Golle, “Revisiting the Uniqueness of Simple Demographics in the US Population”, WPES 2006.
• Machanavajjhala et al., “ℓ-Diversity: Privacy Beyond k-Anonymity”, ICDE 2006；Li, Li & Venkatasubramanian, “t-Closeness”, ICDE 2007；LeFevre, DeWitt & Ramakrishnan, “Mondrian Multidimensional k-Anonymity”, ICDE 2006.
• Datta, Sharma & Sinha, “Provable De-anonymization of Large Datasets with Sparse Dimensions”, POST 2012；Rocher, Hendrickx & de Montjoye, “Estimating the success of re-identifications in incomplete datasets using generative models”, Nature Communications 2019.

Membership inference

• Yeom, Giacomelli, Fredrikson & Jha, “Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting”, CSF 2018.
• Carlini, Chien, Nasr, Song, Terzis & Tramèr, “Membership Inference Attacks From First Principles”, IEEE S&P 2022（LiRA）。
• Salem et al., “ML-Leaks”, NDSS 2019；Watson et al., “On the Importance of Difficulty Calibration in Membership Inference Attacks”, ICLR 2022；Ye et al., “Enhanced Membership Inference Attacks against Machine Learning Models”, CCS 2022；Zarifzadeh et al., “Low-Cost High-Power Membership Inference Attacks”, ICML 2024.
• Choquette-Choo, Tramèr, Carlini & Papernot, “Label-Only Membership Inference Attacks”, ICML 2021；Long et al., “Understanding Membership Inferences on Well-Generalized Learning Models”, 2018；Homer et al., PLoS Genetics 2008（GWAS 汇总数据中的成员推断）。

Memorization, extraction, unlearning

• Feldman, “Does Learning Require Memorization? A Short Tale about a Long Tail”, STOC 2020；Feldman & Zhang, “What Neural Networks Memorize and Why”, NeurIPS 2020；Carlini et al., “The Privacy Onion Effect: Memorization is Relative”, NeurIPS 2022.
• Carlini, Liu, Erlingsson, Kos & Song, “The Secret Sharer”, USENIX Security 2019；Carlini et al., “Extracting Training Data from Large Language Models”, USENIX Security 2021；Carlini et al., “Quantifying Memorization Across Neural Language Models”, ICLR 2023；Nasr et al., “Scalable Extraction of Training Data from (Production) Language Models”, 2023.
• Lee et al., “Deduplicating Training Data Makes Language Models Better”, ACL 2022；Kandpal, Wallace & Raffel, “Deduplicating Training Data Mitigates Privacy Risks in Language Models”, ICML 2022；Duan et al., “Do Membership Inference Attacks Work on Large Language Models?”, COLM 2024.
• Bourtoule et al., “Machine Unlearning”, IEEE S&P 2021（SISA）；Guo et al., “Certified Data Removal from Machine Learning Models”, ICML 2020；Thudi et al., “On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning”, USENIX Security 2022；Hayes et al., “Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy”, 2024.

14. 下一讲预告 · Next lecture

Lecture 10：差分隐私（Differential Privacy）。

本讲的每一个防御都有同一个缺陷：它们只能说明”这些攻击失败了”。去匿名化依赖攻击者的辅助信息，正则化对最脆弱的记录不单调，只返回标签会被多次查询绕过，删掉高风险的人会暴露下一层。第 10 讲换一个问题：能不能给出一个对任何攻击、任何辅助信息都成立的保证？Dwork, McSherry, Nissim & Smith（2006）的答案就是 2022 年幻灯片上的那句话 —— released statistic stays about the same if any individual’s record is removed from the database。我们会从计数查询上的 Laplace 机制开始，到 DP-SGD（梯度裁剪 + 噪声）和 privacy accounting，再回到本讲的工具：用 LiRA 式的成员推断去审计一个 DP 模型，把经验的 TPR / FPR 和 ε 给出的理论上界 $\text{TPR} \le e^\varepsilon\,\text{FPR} + \delta$ 放在同一张图上；最后看 DP 的代价，以及它为什么对 §7 里那些罕见、离群的人代价最大。