Lecture 6 · 缓解偏差 · Mitigating Bias

对应 Maps to：Stanford CS329T Spring 2022 第 4 周的课堂报告主题 “Mitigation in Data”，以及第 5 周（2022-04-26 Datta & Mitchell 的 “Fairness in ML” slides 后半：How can we mitigate bias in ML systems?（pre / in / post-processing）、WAE vs WYSIWYG 两条轴上的方法定位、debiasing word embeddings（neutralize + equalize）、counterfactual data augmentation、upstream vs downstream、adversarial debiasing、feature pruning based on feature-wise bias amplification）。第 5 周阅读材料：Zhang, Lemoine & Mitchell 2018 “Mitigating Unwanted Biases with Adversarial Learning”；Bolukbasi et al. 2016 “Man is to Computer Programmer as Woman is to Homemaker?”；Lu et al. 2018 “Gender Bias in Neural Natural Language Processing” 与 Sun et al. 2019 的综述；Caliskan et al. 2017（WEAT）；Leino et al. 2019 “Feature-Wise Bias Amplification”；Leslie 2020 “Understanding bias in facial recognition technologies”。还对应 HW3 第 2 部分（adversarial training，demographic parity 与 equality of opportunity 两个版本）和第 3 部分（word2vec 的 subspace projection），Lab 4 / Lab 5，以及课程概览里的一张幻灯片 “Example ways to reduce bias”。 配套代码 Code：code/lecture06/（numpy + 一个 PyTorch 脚本；5 个脚本 + 2 个工具模块；03 约 25 秒，其余都在 5 秒以内；导入第 1 讲的 trust_data.py 和 trust_tools.py） 预计阅读时间：3 小时（含运行代码）

0. 本讲导航 · How to read this lecture

0.1 Learning goals

After this lecture you should be able to:

1. 沿着一条 ML 流水线（世界 → 数据 → 标签 → 特征 → 模型 → 决策）说出偏差可能从哪里进来、又在哪里被放大，并用”差距在每一步变成多少”的方式诊断它。
2. Place every mitigation method at one of three intervention points — pre-processing (change the data), in-processing (change the objective), post-processing (change the decision rule) — and say what each one requires: group labels at training time, group labels at decision time, true labels, the ability to retrain.
3. 解释四种代表性方法为什么这样设计：reweighing 的权重为什么是 $P(a)P(y)/P(a,y)$；disparate impact remover 为什么用分位数；协方差约束为什么是凸代理；Hardt et al. 为什么需要随机化。
4. Implement and analyze adversarial debiasing (Zhang et al. 2018): what the adversary sees determines which fairness criterion you get, and what the projection term in the update is for.
5. 读懂一张”所有方法放在一起”的比较表：每个方法修好了它瞄准的指标、同时移动了别的指标；准确率代价主要由”决策时能不能看群体”决定；以及当标签本身有偏时，哪些方法会把偏差当成真相。
6. 在词向量上测量和去除偏见（性别方向、WEAT、hard debiasing、CDA），并解释 “lipstick on a pig”：为什么”测出来没有偏见”往往只说明你用来测的东西被去偏方法直接处理过。

0.2 先修与本讲范围 Prerequisites & scope

第 5 讲回答了”量什么、为什么不能全要”。本讲回答”量出来之后怎么改，改的代价是什么”。我们不重复各个指标的定义，也不重复第 1 讲 §7.1 已经在 oracle 上做过的三种确定性阈值策略；后处理部分直接进入 Hardt et al. 的随机化 equalized odds。

0.3 本讲地图 Map

§1  偏差从哪里进来、在哪里被放大                                   ← 代码 01 演示 1–3
§2  三个干预点：各自需要什么；两种世界观上的定位
§3  Pre-processing：reweighing、disparate impact remover、fair representations、CDA   ← 代码 02 演示 1–2
§4  In-processing：协方差约束、reductions、adversarial debiasing     ← 代码 02 演示 3、代码 03
§5  Post-processing：Hardt et al. 的 equalized odds（ROC 几何 + 随机化）、需要多少数据   ← 代码 04 演示 1–2
§6  一张总表；当标签本身有偏                                       ← 代码 04 演示 3–4
§7  NLP 中的偏见：性别方向、WEAT、hard debiasing、lipstick on a pig、CDA   ← 代码 05
§8  LLM 时代的偏见评估（预告）
§9–13 小结、误区、练习、延伸阅读、下一讲预告

0.4 Notation

0.5 关于数字 · About the numbers

所有数字来自配套代码的实际输出（固定随机种子，同一台机器上重复运行逐字相同）。信贷实验共用三份数据（make_splits()）：训练集 4000 人（和第 1 讲同一份，所以 baseline 就是第 1 讲的 Model B），验证集 20000 人（只给后处理方法设阈值用），测试集 20000 人。测试集取得大，是为了让”方法之间的差别”大于”测量噪声”（§6 给出置信区间）。词向量实验用自己生成的玩具语料。数字只说明机制，不代表任何真实系统。

1. 偏差从哪里进来、在哪里被放大 · Where bias enters, and where it grows

1.1 A pipeline view

Before choosing a fix, locate the problem. Suresh & Guttag (2021) give a useful map of the ML life cycle, naming a kind of bias at each stage. 第 5 讲 §1.7 用 Barocas & Selbst（2016）的机制讲过”歧视从哪里来”；这里换成”流水线的哪一段”，因为修法是按段来的：

Two observations frame the whole lecture. First, the same observed gap can come from different stages, and the right fix depends on which one. Second, some stages don’t just pass bias along — they amplify it.

1.2 沿着流水线看差距（代码 01 演示 1）

在测试集上，把差距一段一段地算出来：

读这张表：

1. 决策环节会放大差距。 即使用完美的、两个群体内部都校准的 oracle 概率，对所有人用同一个阈值，批准率差距（0.161）也比还款率差距（0.141）大。阈值把”概率上的差别”变成”批 / 不批的差别”：group 1 有更多人的概率落在 0.5 下方附近。
2. 模型设定的错误会再放大一次。 不看群体的 baseline 把差距放大到 1.83 倍，并且系统性地低估 group 1（平均低 0.039）、高估 group 0。这就是 aggregation bias：一个 history 系数要同时服务”历史长短很说明问题”的 group 0 和”历史短不代表风险”的 group 1。
3. 修好模型是双赢的。 只加一个群体特征就修好了两个群体的”平均水平”；再加交互项，模型学到不同的 history 系数（group 0：+0.917，group 1：+0.231，和真实世界的 0.8 vs 0.15 方向一致），准确率上升到 0.773（oracle 是 0.775），差距回到和 oracle 一样的 +0.156。如果差距来自模型的错误，这是第一件该做的事。
4. 剩下的 +0.156 是世界的差距加上决策环节的放大。要不要、用什么标准把它也改掉，是第 5 讲讨论的规范性问题 —— 而且”修好模型”本身要求训练和决策时都使用群体标签，这在信贷等领域可能不被允许（§2.2）。本讲后面的大部分方法，都是在回答”不看 $a$、或者只在训练时看 $a$，能做到什么程度”。

1.3 Representation：少数群体越小，“平均”模型越不像它（代码 01 演示 2）

A shared coefficient is a compromise between what each group “wants”, weighted by sample size; the smaller the group, the further the compromise sits from it. 注意同样是 30%，这里的低估（−0.017）比 §1.2 的（−0.039）小 —— 换一份训练集，模型的错误本身也会变（第 1 讲 §8 的提醒）。

2022 年课程概览的幻灯片把 “collect unbiased data” 列为最简单的办法，紧接着写 “not generally possible in practice”。它有时确实可行：Gender Shades 的审计（Buolamwini & Gebru 2018）公开之后，Raji & Buolamwini（2019）的后续审计发现，被点名的几家公司在新版本里都缩小了群体间的错误率差距，深肤色女性上的改进最大。但对信贷这样的场景，“多收集少数群体的数据”往往意味着先给更多人放贷 —— 这本身就是一个决策。

1.4 Bias amplification

Zhao et al. (2017), “Men Also Like Shopping”. In a visual semantic role labeling dataset (imSitu), 66% of “cooking” images had a woman as the agent; a trained model labeled the agent “woman” in 84% of cooking images at test time. The model didn’t just learn the correlation — it exaggerated it. Their fix (RBA) adds corpus-level constraints at inference time so that predicted co-occurrence ratios stay close to the training ratios. 我们的 §1.2 是同一个现象的表格版：还款率差距 0.141 → 决策差距 0.259。

Leino et al. (2019), “Feature-Wise Bias Amplification”. 这篇论文来自 CS329T 的授课老师 Anupam Datta 的组。它把”模型比数据更偏”定义为 $B_D(h) = \mathbb{E}_{(x,y)\sim D}[h(x) - y]$（预测为 1 的比例减去真实为 1 的比例），区分两种来源：

• 不可避免的部分：先验偏斜时，Bayes 最优分类器本身就会过多地预测多数类（特征信息越少越严重）。去掉它就会损失准确率 —— 这和 §1.2 里 oracle 也放大差距是同一件事。
• Feature-wise 的部分：先验平衡时也会出现。训练数据不够时，梯度下降会高估”弱特征”（moderately predictive features）的作用；如果弱特征多数指向同一个类别，模型就会偏向那个类别。这部分可以追溯到具体的特征，因此可以用特征选择来缓解：他们提出 “feature parity”（剪掉指向多数类的弱特征，直到两边数量相等）和 “experts”（只保留影响最大的几个特征）两种算法。

代码 01 演示 3 是论文合成实验的一个简化版：两个类别先验各 50%；两个强特征（方差 1，一个指向类别 1、一个指向类别 0）；$k$ 个弱特征（方差 10，全部指向类别 1）；用最朴素的梯度下降训练逻辑回归（不标准化特征、不正则化，从 0 出发 300 步），重复 30 次取平均：

• 200 个弱特征、100 个训练样本时，模型把 72% 的样本预测为类别 1 —— 先验明明是 50/50。数据增多，放大减弱。
• 剪掉弱特征后 bias 消失、准确率不变（0.756 → 0.755），对应幻灯片上 “high accuracy and low bias” 的说法；但数据足够时（N = 1000）弱特征其实有用，剪掉会损失准确率（0.902 → 0.761）。
• 一个我们没有预想到的结果：在这个玩具里，只把特征减去训练集均值再训练，bias 也消失了，而且准确率高得多（0.961）。没中心化时，弱特征的均值造成的偏移要靠截距抵消，而从 0 出发的梯度下降对截距和其他权重一视同仁，抵消得不够。这说明在这个玩具里放大可以追溯到具体的特征和训练细节；它不说明中心化在一般情况下就能解决问题 —— Leino et al. 分析的是更一般的机制，并在真实数据和 CNN 上做了实验。

The general lesson: a model can be more biased than its data, through the decision rule (thresholding), through misspecification (aggregation), and through how it is trained (feature-wise). Each has a different remedy, and “measure the data, not just the predictions” is how you tell them apart.

2. 三个干预点 · Three intervention points

2.1 What the slides list

课程概览的幻灯片 “Example ways to reduce bias” 只列了三条：

• Collect unbiased data —— “最简单”，但通常做不到；
• Fix the model / change the objective function —— invasive（要动训练过程）、可能损害性能、model-dependent；
• Synthesize unbiased data —— model-agnostic，代表是 counterfactual data augmentation。

第 5 周的 slides 按流水线的阶段把方法分成三类（引用了三篇论文：Zemel et al. 2013 的 fair representations、Woodworth et al. 2017 的 “Learning Non-Discriminatory Predictors”、Hardt et al. 2016）：

• Pre-processing：上调少数群体样本的权重；学习一种让算法”抓不住”群体成员身份的表示（fair representations）。
• In-processing：在损失函数里加一项公平约束（在一个 batch 内计算）；把搜索空间限制在满足独立性条件的函数里（例如”给定真实结果，输出与群体独立”）。
• Post-processing：按群体选阈值以实现 equality of opportunity；对每个群体分别校准，使得”给定预测，群体不再提供额外信息”（sufficiency）。

2.2 每个方法需要什么 · What each method requires

在真实项目里，决定”能不能用”的往往不是准确率代价，而是需要什么：

Three practical constraints follow from this table:

• Legal. In many jurisdictions, explicitly treating people differently by a protected attribute at decision time is itself disparate treatment. US fair-lending law, for example, generally prohibits creditors from considering race or sex in credit decisions — which rules out group-specific thresholds and the DI remover’s group-specific repair in that setting, even when their purpose is to reduce disparate impact. This tension between disparate treatment and disparate impact (第 5 讲 §1.3、§6.5) is why “train-time-only” methods exist. But Lipton, McAuley & Chouldechova (2018) showed that such disparate learning processes reach parity by distorting the model within groups through proxies — we will see exactly this in §4.1.
• Privacy and availability. You can’t use a group label you don’t have. Collecting race or health status is restricted (e.g., special-category data under GDPR), and inferring it is itself risky (第 5 讲 §9.3；第 1 讲 §7.4 的 “fairness → privacy”）。
• Engineering. Post-processing is attractive precisely because it needs only scores — no retraining, no access to the model’s internals, works with a vendor’s black box.

2.3 两条轴：干预点 × 世界观

2022 年的 slides 把方法放在一张二维图上：横轴是 pre / in / post，纵轴是第 5 讲 §1.5 的两种世界观 —— 靠近 WAE（We’re all equal：观测到的差异来自结构性偏差，所以应该追求结果的均等，自然的指标是 demographic parity）还是靠近 WYSIWYG（What you see is what you get：观测如实反映了能力，偏差 = 预测偏离真实结果，自然的指标是基于准确率的错误率）。Slides 上的定位大致是：

这张表的意思是：选方法之前先选立场。如果你相信差距来自有偏的标签或特征（WAE），追求 demographic parity 的方法是在纠正测量；如果你相信观测是准确的（WYSIWYG），同样的方法是在强行抹掉真实的差异。§6.2 用一个”标签有偏”的实验让这个选择变得具体。

3. Pre-processing：改数据

3.1 Reweighing（Kamiran & Calders 2012）

Idea. Don’t change any record; change how much each record counts. Give every (group, label) cell the weight

Why this formula. The weighted joint distribution is $P_w(a, y) \propto w(a, y)\,P(a, y) = P(a)\,P(y)$ — a product, so in the reweighted data the label is independent of the group. The average weight is $\sum_{a,y} P(a,y)\,w(a,y) = \sum_{a,y} P(a)P(y) = 1$, so the effective sample size is unchanged. Under-represented cells (here: group 1 repayers) get weights above 1. Kamiran & Calders 也提出了 massaging（把边界附近的一些样本的标签翻过来）和 resampling，思路相同：让训练数据”看起来”满足 independence。

代码 02 演示 1 的权重：group 0 的违约 / 还款样本是 1.132 / 0.947，group 1 是 0.787 / 1.149；加权后两个群体的还款率都是 0.676。效果：

同样的权重，交给不同的模型，效果差别很大。不看群体的模型没有”群体”这个旋钮，只能通过和群体相关的特征（history、income）间接调整，而这些系数还要服务于预测本身，所以挪不了多少（+0.259 → +0.232）。看群体的模型只需要把 group 1 的截距抬高：加权后的最优截距正好让两个群体的还款率”相同”，批准率差距几乎一步降到 +0.036。代价：FPR 差距反向（group 1 里会违约的人更容易被批准），PPV 差距扩大到 +0.136，group 1 内 ECE 变成 0.077 —— 模型被故意训练成对 group 1 过于乐观。So “model-agnostic” pre-processing is only as effective as the model’s ability to express the change.

3.2 Disparate impact remover（Feldman et al. 2015）

Idea. If a feature is distributed differently across groups, it can act as a proxy. Repair each feature so that its distribution is the same in every group, while preserving each person’s rank within their own group:

where $F_a$ is the feature’s CDF in group $a$ (“I am at the 70th percentile of my group”), $Q_a$ is its quantile function, and $\lambda \in [0, 1]$ is the repair level. With $\lambda = 1$, someone at the 70th percentile of group 1’s history distribution and someone at the 70th percentile of group 0’s both get the same value.

Why quantiles. Feldman et al. connect disparate impact to predictability: if the protected attribute can’t be predicted from the features (balanced error rate close to 1/2), no classifier trained on them can have large disparate impact（第 5 讲 §6.2 讲了这个”可认证”的论证）。Making every marginal identical across groups attacks predictability directly, and rank preservation keeps as much within-group information as possible. 代码 02 演示 2：

• λ = 1 时，任何特征、甚至所有特征合起来（“群体探针”：用修复后的特征训练逻辑回归去预测群体）都看不出群体了，批准率差距消失。在我们的世界里特征在给定群体时相互独立，所以只修边缘分布就够了；特征之间有群体特有的相关结构时，联合分布仍可能泄露群体。
• 准确率只降了 0.009，代价在别处：FPR 差距反向到 −0.130，PPV 差距 +0.154，group 1 内 ECE 0.107。
• λ 是一个连续的旋钮，不是开关：λ = 0.5 时差距已降一半，代价小得多。
• 它在决策时需要群体标签（要用对应群体的 $F_a$ 修复每个申请人的特征）。这是它准确率代价小的原因，也是它在很多场景里不能用的原因。

3.3 Fair representations（简介）

Zemel et al. (2013), “Learning Fair Representations” (LFR), learn a mapping from $x$ to a probabilistic assignment over a small set of prototypes, trained on a weighted sum of three losses: (i) statistical parity of the representation across groups, (ii) reconstruction of $x$, (iii) prediction of $y$. Any downstream model trained on the representation inherits its parity. Later work replaced the hand-designed parity term with an adversary that tries to predict $a$ from the representation (Edwards & Storkey 2016; Madras et al. 2018, LAFTR) — the same idea as §4.3, applied to an intermediate layer instead of the output. 好处是”一次去偏、处处可用”（数据发布者去偏，下游任何人用）；坏处是表示学习本身需要群体标签，而且下游任务换了，原来的权衡未必还合适。

3.4 Counterfactual data augmentation（预告）

2022 年 slides 把 CDA 当作 “synthesize unbiased data” 的代表：把每个句子里的性别词换成另一个性别（he → she、king → queen、John → Mary），把换过的句子加进训练数据，再训练模型（Lu et al. 2018，用在共指消解 coreference resolution 上，slides 展示了它对几乎所有职业都缩小了性别偏差）。它是 model-agnostic 的，只需要一张替换表。§7.5 会看到它修了什么、没修什么。

4. In-processing：改目标函数

4.1 把约束写进目标函数（Zafar et al. 2017 的协方差代理）

Idea. Demographic parity is a constraint on decisions, which are discontinuous in $\theta$ — hard to optimize. Zafar, Valera, Gomez Rodriguez & Gummadi (2017) replace it with a convex proxy: the covariance between the group label and the signed distance to the decision boundary,

It is linear in $\theta$, so the constraint $|c^\top\theta| \le \epsilon$ is convex, and its penalty form $\frac{\mu}{2}(c^\top\theta)^2$ keeps logistic regression convex: Newton’s method still works, the Hessian just gains a term $\mu\,c c^\top$（见 mitigation_tools.fit_logistic）。For binary $a$, zero covariance means equal mean logits across groups — close to, but not the same as, equal approval rates. Restricting the sum to $y = 1$ gives a proxy for equal opportunity.

代码 02 演示 3（只在训练时需要群体标签）：

和 DI remover（λ = 1 时准确率 0.761）比，协方差惩罚贵得多（0.718）。原因在系数里：

为了让两个群体的平均分数相同、而决策时又不能看群体，模型只能改所有人共用的系数：history 的权重从 +0.642 翻成负数，income 的权重减半 —— 因为”历史长、收入高”的多半是 group 0。对每个群体内部，这是一个更差的排序：group 0 里历史长、确实更可靠的人反而被扣分。This is precisely the mechanism Lipton et al. (2018) warned about: a disparate learning process achieves parity between groups by using proxies to treat people differently within groups. 决策时能看群体的方法（DI remover、按群体阈值）不需要这样扭曲，所以代价小 —— 但它们在法律上更难被接受。这是一个真实的两难，不是技术能消解的。

4.2 Reductions（Agarwal et al. 2018，简介）

Covariance proxies only work for models where the proxy is tractable. Agarwal, Beygelzimer, Dudík, Langford & Wallach (2018) make fair classification work with any learner that can do cost-sensitive classification:

1. Write the fairness constraint (DP or EO) as linear constraints on the classifier’s conditional moments, $M\mu(Q) \le c$, over randomized classifiers $Q$ (distributions over models).
2. Form the Lagrangian $L(Q, \lambda) = \text{err}(Q) + \lambda^\top(M\mu(Q) - c)$ and treat it as a two-player game: the $\lambda$-player runs exponentiated gradient; the $Q$-player best-responds, and for fixed $\lambda$ the best response is a cost-sensitive classification problem — i.e., ordinary training with reweighted, relabeled examples.
3. The output is a mixture of the classifiers found along the way, with guarantees on accuracy and constraint violation.

所以 reductions 把”公平训练”归约成一系列”加权的普通训练”（Fairlearn 工具包里的 ExponentiatedGradient 就是它）。注意两点：输出是随机化的分类器（和 §5 的 Hardt et al. 一样）；它只在训练时需要群体标签。练习 2 和代码 README 的”动手改一改”让你思考它和 reweighing 的关系。

4.3 Adversarial debiasing（Zhang, Lemoine & Mitchell 2018）

The game. A predictor maps $x$ to a prediction $\hat y$ (logit $z$). An adversary tries to predict the group $a$ from what the predictor outputs. The predictor is trained to predict $y$ well and to make the adversary fail. What the adversary gets to see decides which criterion you are enforcing:

HW3 的两个编程题正是 DP 版本（对手只拿到分类器的预测）和 EOpp 版本（对手还拿到真实类别）。

The update. With predictor loss $L_P$ and adversary loss $L_A$, the adversary’s weights $U$ follow $-\nabla_U L_A$ as usual. The predictor’s weights $W$ move along

• The last term pushes $W$ to increase the adversary’s loss, with weight $\alpha$.
• The middle term removes the component of the predictive gradient that lies along $\nabla_W L_A$. Without it, the predictor could take steps that improve prediction and incidentally help the adversary; the projection forbids moving in any direction that helps the adversary. In the paper’s words, without the projection term “it is possible for the predictor to end up helping the adversary”.

The paper proves that, under strong assumptions (convexity/concavity, an adversary that is optimal and can’t beat $H(A)$ or $H(A \mid Y)$), the equilibrium satisfies the corresponding independence. Neural networks don’t satisfy those assumptions, so in practice the result must be audited with an independent measure — here, the grpAUC of the scores. For the UCI Adult experiment they used a logistic-regression predictor and an adversary $\hat z = w^\top[s,\ s y,\ s(1-y)] + b$ with $s = \sigma\big((1 + |c|)\,z\big)$, trained with Adam, $\alpha$ growing like $\sqrt t$ and the step size shrinking with $t$; they report that without this schedule training often got stuck. 代码 03 沿用这个结构（线性预测器，从普通模型开始，先单独训练对手，再交替训练）。我们自己的经验与之一致：不加 $\sqrt t$ 的 α 增长和步长衰减时，EO 版本会震荡，准确率掉到 0.6–0.67。

代码 03 演示 1（DP 版本，扫描 α；$H(A) = 0.6123$ 是”只知道 group 1 占 30%“的对手能达到的 log loss）：

演示 2（EO 版本：对手多看一个 $y$）：

演示 3（投影项的作用；以及和协方差惩罚的比较）：

What to take away:

1. The adversary’s loss is the fairness meter. As α grows, the adversary’s loss rises to $H(A)$ — it can no longer beat the base rate — and the scores’ grpAUC falls to ~0.52. The DP gap goes to +0.010, with the familiar costs: accuracy 0.719, PPV gap +0.121, ECE in group 1 0.100. TPR and FPR gaps turn negative: DP doesn’t care whose errors increase.
2. EO is a different target, not a weaker DP. With the label visible to the adversary, both within-label grpAUCs reach ~0.51 at α = 3 and the TPR/FPR gaps shrink to about ±0.02, but the DP gap stays at +0.041. Under exact equalized odds with common rates $(t, f)$, the approval gap must equal $(p_0 - p_1)(t - f)$ — positive whenever base rates differ（练习 4）。
3. The projection matters. At α = 1, removing it leaves a DP gap of +0.155 instead of +0.060; at α = 10 the difference shrinks (+0.032 vs +0.010). It lets the same α do more debiasing.
4. Two algorithms, one destination. For an unaware linear model and DP, the adversarial game (PyTorch, Adam) and the covariance penalty (numpy, Newton) land at nearly the same coefficients — history flipped negative, income roughly halved — and nearly the same metrics. 对抗方法的价值不在线性模型上，而在于它对任何可微模型都适用：换成深度网络只需要改一行（README 的”动手改一改”）。
5. α is a hyperparameter with a cost curve. At α = 10 the EO version’s group-0 ECE climbs to 0.098 and accuracy to 0.708. 要像选正则化强度一样，在验证集上画出”准确率 vs 差距”的曲线再选。

5. Post-processing：改决策规则

5.1 为什么需要随机化

第 1 讲 §7.1 在 oracle 上用确定性的按群体阈值实现了 demographic parity 和 equal opportunity：每个约束只涉及一个量（批准率或 TPR），调一个阈值就能对上。Equalized odds 要求 TPR 和 FPR 同时相等 —— 两个约束，每个群体只有一个阈值，一般对不上。Hardt, Price & Srebro (2016) 的解决办法是允许随机化的决策规则。

5.2 ROC geometry（代码 04 演示 1）

Fix a score $R$ (here: the unaware baseline). For group $a$, the rule “approve if $R \ge t$” gives one point $(\text{FPR}_a(t), \text{TPR}_a(t))$; sweeping $t$ traces the ROC curve. Now the key observation: randomizing between two thresholds — use $t_1$ with probability $q$, else $t_2$ — gives exactly the point $q\,P_1 + (1-q)\,P_2$ on the segment between them. Approving everyone at random with probability $r$ gives the diagonal point $(r, r)$. So the set of $(\text{FPR}, \text{TPR})$ pairs achievable for group $a$ is the convex hull $H_a$ of its ROC points — the region between its upper concave envelope and the diagonal.

Equalized odds requires both groups to sit at the same point $(x, y)$, so the feasible set is $H_0 \cap H_1$: everything below the lower of the two envelopes. Among those points we want the one with the smallest expected error,

with $\pi_a$ the group shares and $p_a$ the base rates. The objective is linear and the feasible set is a convex polygon, so the optimum is at a vertex: either a vertex of one envelope or a crossing point of the two. 这就是论文里的线性规划（LP）的二维几何版本：变量是每个群体”在每个分数区间里批准的概率”，约束是两个群体的 TPR、FPR 相等，目标是期望损失。EqualizedOddsPostprocessor.fit 把候选顶点逐个比较。

Once the target point is chosen, each group needs a rule that reaches it:

• If the point lies on the group’s envelope, mix two adjacent hull thresholds $t_{hi} > t_{lo}$: approve above $t_{hi}$, reject below $t_{lo}$, and approve with probability $q$ in between.
• If it lies below the envelope (the group “could do better”), additionally mix with the diagonal: with probability $\lambda$ use the curve rule, otherwise approve at random with probability $r$. The better-served group is deliberately degraded until it matches the other one.

验证集（20000 人）上的结果：

• group 0：AUC 0.819，凸包 59 个顶点，阈值 0.5 的工作点 (FPR, TPR) = (0.579, 0.926)；
• group 1：AUC 0.781，凸包 39 个顶点，工作点 (0.327, 0.747)；
• EO 目标点：(0.520, 0.884)，验证集期望错误率 0.247（单一阈值 0.5：0.239）。它恰好是 group 1 包络的一个顶点 —— group 1 用确定性阈值 0.359；group 0 的目标点在它的包络下方：以概率 λ = 0.922 使用规则”score ≥ 0.570 批准，0.537–0.570 之间以概率 0.529 批准”，否则不看分数、以概率 0.702 批准。

  TPR
   1.0 |                                              001111111111111
       |                                   A011111111111        ...
       |                            000*11111                ...
       |                      000011111                   ...
       |                  00001111                     ...
   0.8 |               000 1B1                      ...
       |            000 111                      ...
       |          00 111                      ...
       |        00 11                      ...
       |       0  1                     ...
   0.5 |      0 11                   ...
       |    00 1                  ...
       |      1                ...
       |   0 1              ...
       |  0 1            ...
   0.2 |   1          ...
       | 01        ...
       |        ...
       | 1   ...
       |  ...
   0.0 |1.
       +-------------------------------------------------------------
        0.0                         0.5                          1.0   FPR
  0 / 1 = 两个群体的 ROC 上凸包；. = 对角线；A / B = 阈值 0.5 时 group 0 / 1 的工作点；* = EO 目标点

测试集（20000 人）上，和确定性的按群体阈值比较：

• TPR、FPR 差距同时降到约 0；准确率代价 0.015，比确定性阈值大（多了一个约束，还要给 group 0 加噪声）。
• 个体层面的彩票。 group 0 申请人的决策有 10.4% 的概率由硬币而不是分数决定：分数最高的人也有 2.3% 的概率被拒，分数最低的人有 5.5% 的概率被批。两个分数相同的人可能一个被批、一个被拒。这满足了群体层面的 separation，却和 individual fairness（“相似的人被相似地对待”，Dwork et al. 2012；第 5 讲 §5）冲突，也很难向被拒的人解释。
• Leveling down. group 0 的表现被故意拉低，而 group 1 的表现一点也没有提高。如果目标是”让 group 1 得到更好的决策”，这不是你想要的。后处理只能在给定的分数上做到最好；要真正改善 group 1 的决策，需要让分数对 group 1 也有区分力（更好的特征和数据，§1.2）。Woodworth et al.（2017）还证明了：在某些分布上，先学一个普通模型再做后处理，可能比直接在训练时施加 equalized odds 差得多。

For equal opportunity alone (only TPR), Hardt et al. show that with a continuous score, group-specific deterministic thresholds suffice — matching the EOpp row above. Pleiss et al. (2017) add a negative result for the slides’ other post-processing suggestion (“calibrate each group separately”): calibration within groups is compatible with equalized odds only in degenerate cases, and the relaxed version they propose works by randomly withholding information from part of one group.

5.3 后处理需要多少数据？（代码 04 演示 2）

凸包是在有限的验证集上估计的。每种验证集大小重复 20 次：

在拟合用的数据上差距永远恰好为 0；在新数据上，验证集小的时候平均还剩 0.02–0.04 的差距 —— 一部分是测试集本身的抽样噪声，一部分是凸包”挑中了碰巧好的顶点”。公平约束是在有限样本上满足的：报告”后处理后差距为 0”时，要说明是在哪份数据上，并在独立数据上重新测量（第 1 讲 §8.1）。

6. 一张总表 · Every method, one table

6.1 所有方法，同一个测试集（代码 04 演示 3，对抗去偏来自代码 03）

（后处理的行没有 ECE：分数没变，变的是每个群体用哪个阈值 —— 同一个分数在两个群体里对应不同的决策，这本身就是 sufficiency 被破坏的样子，体现在 PPV gap 上。）

95% bootstrap 置信区间（重抽测试集 400 次）的半宽在 0.011–0.025 之间，例如 baseline 的 DP gap 是 [+0.243, +0.272]、Hardt EO 的 TPR gap 是 [−0.011, +0.011]、协方差惩罚 DP 的 PPV gap 是 [+0.103, +0.137]。它们只反映测试集的抽样噪声，不包括换一份训练集带来的变化（§1.3 已经看到那部分也不小）。

Reading the table:

1. Every method fixes the metric it targets and moves the others. DP-type methods (reweighing +a, DI remover λ = 1, DP penalty, adversarial DP, DP thresholds) drive the approval gap to ~0, and in exchange the FPR gap flips sign and the PPV gap grows from +0.049 to +0.12–0.15. EOpp-type methods zero the TPR gap but leave an FPR gap. Hardt EO zeroes both error-rate gaps and still widens the PPV gap. No row zeroes DP, TPR, FPR and PPV gaps at once — the impossibility theorems of 第 5 讲 §4, now as numbers.
2. The accuracy cost is mostly about access to $a$ at decision time. Methods that may look at the group when deciding cost 0.002–0.015; the train-time-only methods that actually close the gap (penalties, adversarial training) cost 0.04–0.05, because they must bend shared coefficients (§4.1).
3. “Fix the model” is the only row that improves accuracy — and it removes only the model’s own error, leaving the oracle’s gap.
4. Gaps smaller than the CI half-width (~0.02) don’t distinguish methods. “Method A has DP gap 0.006 and method B 0.017” is not evidence that A is fairer.

6.2 当标签本身有偏（代码 04 演示 4）

所有用到 $y$ 的方法 —— 修好模型、reweighing、EO / EOpp 的约束、Hardt 后处理 —— 都默认标签是我们真正关心的东西。如果不是呢？演示 4 构造了一个最简单的标签偏差：历史记录里，group 1 中随机 20% 的真实还款者被记成了违约（第 5 讲 §7 的 “flip” 机制）。group 1 记录下来的还款率是 0.462，真实是 0.571。所有方法都只能用记录的标签训练和设阈值：

• 标签偏差让 baseline 更偏了（真实 TPR 差距从 +0.183 变成 +0.265）。
• “修好模型”在这里有害：它把有偏的标签学得更准，批准率差距 +0.371、真实 TPR 差距 +0.306，都比 baseline 大，按真实结果算的准确率却几乎没变（0.763 → 0.764）。§1.2 的”双赢”只在 WYSIWYG 成立时成立。
• Hardt EO 在记录的标签上”完成了任务”；在真实结果上，TPR 差距仍约为 0，FPR 差距却是 +0.057。原因和第 5 讲 §7.4 一样：随机误记让”记录为还款”的人仍是真实还款者的随机子集（TPR 不受影响），却把真实还款者混进了”记录为违约”的人里（FPR 被扭曲）。哪个指标被骗，取决于标签偏差的结构。
• 不用标签定义约束的 DP 方法不受这种扭曲：批准率差距本来就不依赖标签，而在这个世界里它在真实结果上的 TPR / FPR 差距也很小（−0.029 / +0.002）。这是 WAE 立场的论据：不信任标签，就不要让公平指标依赖它。反过来，如果标签可信，DP 会强行抹掉真实的差异（§6.1 里 DP 方法的 PPV 代价）。

So the choice between methods is not a technical one. It encodes a belief about where the bias comes from, which is exactly the WAE / WYSIWYG question of §2.3. The 2022 slides call this “upstream vs downstream”: upstream fixes (data, labels, representations) address the cause if you know it; downstream fixes (thresholds) are cheap and model-agnostic but can only rearrange the outcomes of whatever scores and labels they are given.

6.3 一份选择清单 · A checklist

1. 差距从哪一段来？ 先像 §1.2 那样一段一段地量。模型的错误（aggregation、feature-wise amplification）先修模型；数据的代表性问题先考虑补数据和 reweighing。
2. 标签可信吗？ 不可信时，所有依赖标签的约束和”修好模型”都会继承标签偏差（§6.2）。
3. 决策时能用群体标签吗？ 不能 → 只剩 train-time 方法（reweighing、约束、对抗训练），要接受更大的准确率代价和群体内部的扭曲；能 → 后处理和 DI remover 更便宜，但要准备好法律和解释上的论证。
4. 哪种错误对谁伤害更大？ 这决定瞄准 DP、EOpp 还是 EO（第 5 讲 §10）。
5. 报告所有指标，带置信区间，并在独立数据上重新测量（§5.3、§6.1）。写进 model card（第 1 讲 §8.4）。

7. NLP 中的偏见 · Bias in NLP

7.1 为什么 NLP 不一样

信贷模型的偏见是 allocative 的：谁得到了贷款。语言模型和词向量里更多的是 representational harm（第 5 讲 §1.6）：刻板印象被编码、被补全、被放大，而且这些表示会被成千上万个下游系统复用。词向量是语料统计的压缩：一个词的向量由它和哪些词一起出现决定。所以第一个问题是 —— 语料里的刻板印象会以什么几何形式出现在向量里？

代码 05 用一个”知道埋了什么偏见”的玩具语料来回答（toy_embeddings.py）：32 个职业分属 8 个话题，每个话题里正好 2 个偏男性、2 个偏女性的职业（这样”话题相似”本身不能预测刻板印象）；职业句里代词的性别按设定的 $p_\text{male}$ 抽取；另外可选地加入性别化的语境词（dress / beard）、刻板化的描述词（care / gentle vs strong / lead）、以及”男性名字更常和 career 词一起出现”的生活句。词向量用 PPMI + SVD 训练（Levy & Goldberg 2014 证明了 word2vec 的 SGNS 隐式地分解一个平移过的 PMI 矩阵；Levy, Goldberg & Dagan 2015 把 SGNS 的一些技巧移植到了 PPMI 上），30 维，130 个词。

7.2 The gender direction（Bolukbasi et al. 2016；代码 05 演示 1）

Idea. Take definitional pairs — (he, she), (man, woman), (king, queen), … — subtract each pair’s mean from both words, stack the differences, and run PCA. If the differences mostly point the same way, the first principal component $g$ is a gender direction. The direct bias of a gender-neutral word $w$ is its projection $w \cdot g$.

In our corpus the first component explains 78.4% of the variance and the second 20.6% — gender is mostly one direction, not entirely (Bolukbasi et al. also found the first component clearly dominant in word2vec, which justified using $k = 1$). The occupations’ projections reproduce the corpus statistics almost perfectly:

32 个职业的投影与 $p_\text{male}$ 的相关系数 r = +0.995。The embedding didn’t invent the stereotype; it recorded the corpus — which is exactly why embeddings trained on real text record real stereotypes.

Analogies, and a measurement caveat. The famous title comes from the analogy query “man : computer programmer :: woman : ?” answered by $\arg\max_w\, w \cdot (b - a + c)$ (3CosAdd). Standard implementations (including HW3’s compute_analogy) exclude the three input words from the answers. In our toy, “he : king :: she : ?” returns queen with exclusion and king without it — $b$ itself is usually the nearest word, which is why exclusion is the default. But Nissim, van Noord & van der Goot (2020) point out the flip side: exclusion makes “he : doctor :: she : doctor” impossible, so some celebrated biased analogies are partly artifacts of the query. In our topic-balanced toy, occupation analogies return same-topic occupations of the same gender lean (programmer → engineer, nurse → midwife), so the analogy test misses a stereotype that the projection shows with r = 0.995. Different instruments, different verdicts.

7.3 WEAT（Caliskan, Bryson & Narayanan 2017；代码 05 演示 2）

WEAT 模仿心理学里的内隐联想测验（IAT）。给定两组目标词 $X, Y$（如 career / family）和两组属性词 $A, B$（如男性 / 女性名字）：

其中 $(X_i, Y_i)$ 遍历 $X \cup Y$ 的所有等大小划分（单侧置换检验）。$X$、$Y$ 各 6 个词时一共 924 种划分，所以 p 最小是 1/924 ≈ 0.001。

Two lessons about reading WEAT:

• The effect size saturates. With 6 + 6 words, $d$ cannot exceed $2/\sqrt{12/11} \approx 1.915$ (all $s(x)$ equal, all $s(y)$ equal). Our main corpus hits the ceiling with both attribute sets, so $d$ no longer distinguishes “strong” from “very strong” — look at $S$. Caliskan et al.’s career/family test with male/female names on GloVe gave an effect size of about 1.8, also near this ceiling.
• $d$ is standardized, so it can be large when nothing is there. In the control corpus (no career/family association planted), $d = 0.78$ — “large” by Cohen’s convention — while $S$ is about 1/70 of the main corpus’s and $p = 0.095$. Read $S$, $d$ and $p$ together, and use a control. Note also that the permutation test treats words as the random units: it asks whether this split of these 12 words is special, not whether another corpus would show the same association. 对 WEAT 作为偏见测量的更多批评见 Ethayarajh, Duvenaud & Hirst（2019）。

7.4 Hard debiasing：neutralize + equalize（代码 05 演示 3）

Bolukbasi et al. 的两步（HW3 第 3 部分要你实现的正是这两个函数）：

• Neutralize：对每个”不该有性别”的词（不在 gender-specific 词表里），去掉它在 $g$ 上的分量并归一化：$w \leftarrow (w - (w\cdot g)g)/\lVert w - (w\cdot g)g\rVert$。
• Equalize：对每一对（he, she）、（king, queen）……，令 $\mu = (w_1 + w_2)/2$，$\nu = \mu - (\mu\cdot g)g$，然后 $$w_i \leftarrow \nu + \sqrt{1 - \lVert\nu\rVert^2}\;\frac{(w_i\cdot g)g - (\mu\cdot g)g}{\lVert (w_i\cdot g)g - (\mu\cdot g)g\rVert} .$$ 两个词只在 $g$ 方向上不同、在其余方向上完全相同，所以任何中和过的词到它们的距离完全相等。

为什么需要 equalize（HW3 书面题 17）？只 neutralize 时，“babysit”到 grandmother 和 grandfather 的距离仍可能不同 —— 因为这两个词在 $g$ 之外的方向上也不同。Equalize 把那部分差别去掉。

The utility checks are untouched, and the projection is exactly zero. Now look at the two WEAT columns. With the definitional words as attributes, WEAT is exactly zero — but that is by construction: equalize makes every neutralized word equidistant from each pair, so every $s(w)$ is zero（练习 5）。With names as attributes, WEAT barely moves: the names were never equalized, and most of the career–John association isn’t along $g$. “After debiasing, WEAT is 0” means nothing until you say which attribute words you used. A test that shares its word list with the debiasing method can hardly fail.

7.5 Lipstick on a pig（Gonen & Goldberg 2019）与 CDA（代码 05 演示 4）

The claim. Gonen & Goldberg (2019), “Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them”, argued that the gender direction is a measure of bias, not its substance. After hard debiasing (Bolukbasi) or GN-GloVe, they took the 500 most male-biased and 500 most female-biased words (by original projection) and found:

• Clustering the debiased vectors into two groups still matched the original male/female split 92.5% of the time (hard-debiased; 99.9% before) and 85.6% (GN-GloVe).
• An RBF-SVM trained on 1,000 of the 5,000 most biased words predicted the original bias of the other 4,000 with 88.88% accuracy after hard debiasing (98.25% before).
• A word’s original bias still correlated with how many of its neighbors were male-biased (for professions, r = 0.606 after hard debiasing vs 0.747 before).

The bias lives in the geometry of the “neutral” words — who their neighbors are — not only in their projection on $g$.

Our version. We check whether a weak linear probe can still read each occupation’s original lean from its vector. To avoid confounding with topic, the probe is trained on 7 topics and tested on the held-out 8th (nearest-centroid classifier, 32 predictions). Its noise floor comes from shuffling labels within topics: mean ≈ 0.49, 95th percentile ≈ 0.69. We also report 2-means clustering, as Gonen & Goldberg did. Three corpora, from “gender appears only through swappable words” to “gender also appears through words no swap list covers”:

• Hard debiasing never lowered the probe’s accuracy, in any corpus — even in W1, where gender enters only through pronouns and names. Gender here is not one-dimensional (the second principal component carried about 21% in §7.2), and names carry gender information that isn’t on $g$. Projection removes one direction; the stereotype is linearly readable from the rest. 这就是 “lipstick”：投影归零让偏见在一个测量下”消失”，却没有从表示里消失。
• CDA worked only in W1 (probe 0.53, at the floor). There every gendered token is in the swap list, so the augmented corpus is exactly symmetric under he ↔ she, John ↔ Mary. In W2 the nurse still co-occurs with dress in both the original and the swapped sentence, so the stereotype survives. CDA removes the co-occurrences its swap list covers, and nothing else. On the other hand, CDA (with names in the swap list) zeroes the names-WEAT exactly, which hard debiasing does not: the two methods fix different things.
• Clustering is a weaker test. 2-means finds only the dominant two-way structure; here topic dominates, so clustering says ~0.5 for almost everything — including the original W1 and W2 embeddings, whose bias the probe reads perfectly. If you only cluster, you will declare biased embeddings clean.
• Caveat. These corpora were designed by us; they show mechanisms (bias can live outside the gender direction and outside the swap list), not how much residual bias real embeddings have. The real-data evidence is Gonen & Goldberg’s.

7.6 Intrinsic vs extrinsic，以及嵌入之外

2022 年 slides 在这里放了一页 “Caveat: intrinsic bias metrics do not correlate with application bias”，问”减小数据 / 词向量的偏见，会减小应用里的偏见吗？“，答案是 “NLP: almost certainly NO”。这指的是 Goldfarb-Tarrant et al.（2021）的发现：WEAT 这类内在（intrinsic）指标和下游任务（如共指消解、仇恨言论检测）上的外在（extrinsic）偏差之间几乎没有相关性。结合 §7.4–7.5：一个去偏方法可能把内在指标降到 0、而下游偏差纹丝不动。所以缓解要在应用层面评估。

嵌入之外的方法（Sun et al. 2019 的综述把它们按”数据 / 训练 / 推理”组织，和本讲的三个干预点一一对应）：

• 数据：CDA（Lu et al. 2018）；把名字也纳入替换（Maudslay et al. 2019 的 counterfactual data substitution）；平衡数据集。
• 训练：对抗学习 —— Zhang et al.（2018）在同一篇论文里也用对手去掉了词类比任务里的性别信息；带约束的训练。
• 推理：Zhao et al.（2017）在推理时加语料级约束，抑制 §1.4 的 bias amplification。
• 评估：WinoBias（Zhao et al. 2018）和 Winogender（Rudinger et al. 2018）用”只差一个代词”的句子对测试共指消解系统 —— 这正是 slides 上 “Bias in coreference resolution” 那几页的例子：系统把 “the physician … he” 连对、把 “the physician … she” 连错。

8. LLM 时代的偏见评估（预告） · Bias evaluation for LLMs

Generative models shift the question again: the output is open-ended text, so “the metric” is a judgment about many possible continuations. The classic ideas carry over in recognizable forms:

• Counterfactual pairs (WinoBias-style) become benchmarks for QA and generation: BBQ (Parrish et al. 2022) asks questions whose answer is unknown in an ambiguous context and stated in a disambiguated one, and measures how often the model falls back on a stereotype; StereoSet (Nadeem et al. 2021) and CrowS-Pairs (Nangia et al. 2020) compare a model’s likelihood of stereotypical vs anti-stereotypical sentences; BOLD (Dhamala et al. 2021) scores open-ended generations about different groups. Blodgett et al. (2021) audited several such benchmarks and found many of their test pairs to be ambiguous or not about stereotypes at all — the measurement lesson of §7.2–7.3 again.
• Name-swap audits are CDA used as a test: send the same résumé, loan application or medical question with only the name changed, and compare outcomes.
• Mitigation happens at every stage: pre-training data filtering (pre-processing), fine-tuning and RLHF with fairness-related preferences (in-processing), and system prompts or output filters (post-processing). Ganguli et al. (2023) found that sufficiently large RLHF-trained models reduce stereotyped answers on BBQ substantially when simply instructed to avoid relying on stereotypes — a post-processing lever that didn’t exist for classifiers.
• And the judges are models too: when bias is scored by an LLM judge, the judge must be validated first (第 1 讲 §8.3；第 11 讲).

第 11–13 讲会回到这些问题：怎样在没有标准答案时评估、怎样验证评估器、怎样让模型说”我不确定”。

9. 小结 · Summary

Six takeaways:

1. Diagnose before you mitigate. Measure the gap at each stage of the pipeline. Gaps caused by model error (aggregation, feature-wise amplification) should be fixed in the model, which can raise accuracy.
2. Every intervention point has a price in requirements: whether you need the group label at training time, at decision time, trustworthy labels, the ability to retrain. These often decide feasibility more than accuracy does.
3. Every method fixes the metric it targets and moves the others. No method zeroes all gaps; the impossibility theorems show up in every comparison table.
4. Methods that can’t see the group at decision time pay more and distort more — they reach parity by bending shared coefficients against proxies, which treats people differently within groups.
5. Choosing a method encodes a belief about where bias comes from (WAE vs WYSIWYG). With biased labels, the “most accurate” fix makes things worse.
6. Debiasing a representation can hide bias from a metric without removing it. Test with instruments independent of the method (held-out probes, downstream tasks), report what you measured with, and evaluate at the application level.

10. 常见误区 · Common pitfalls

• “We removed the protected attribute, so the model can’t discriminate.” Proxies remain (第 5 讲 §6), and removing $a$ can even increase the gap when the gap comes from aggregation bias (§1.2: adding $a$ shrank the gap from +0.259 to +0.170).
• “Reweighing is model-agnostic, so it works the same for any model.” The same weights cut the gap by 0.027 for an unaware model and by 0.134 for an aware one (§3.1).
• “The penalty drove the DP gap to zero at a small accuracy cost, so nobody lost.” Look inside groups: the history coefficient flipped sign, penalizing reliable group-0 applicants with long histories (§4.1).
• “The adversary’s accuracy dropped to chance, so the predictions carry no group information.” Only for that adversary. Audit with an independent measure (grpAUC, a fresh probe); a weak adversary can be fooled.
• “Post-processing equalized the error rates.” On the data it was fitted on, always. On new data with a small validation set, gaps of 0.02–0.04 remain (§5.3); and randomized rules put a lottery inside one group (§5.2).
• “Equalized odds failed because the approval gap isn’t zero.” EO implies an approval gap of $(p_0 - p_1)(t - f)$ when base rates differ. DP and EO are different targets.
• “After debiasing, the projection and WEAT are zero, so the embedding is unbiased.” Equalize makes the definitional-word WEAT zero by construction; names and held-out probes still find the stereotype (§7.4–7.5).
• “Our intrinsic bias metric improved, so the application is fairer.” Intrinsic and extrinsic bias are weakly related (§7.6). Measure where the harm happens.

11. 练习 · Exercises

练习 1（reweighing 的推导）. (a) Show that with $w(a, y) = P(a)P(y)/P(a, y)$, the weighted joint distribution satisfies $P_w(a, y) = P(a)P(y)$ and the average weight is 1. (b) 考虑一个只有”每个群体一个截距”的模型（没有其他特征），用加权的 log loss 训练。证明它对两个群体给出的预测概率都等于整体还款率 $P(Y=1)$，所以批准率完全相同。(c) 用 (b) 解释 §3.1 里”看群体的模型 + reweighing”几乎一步到位、而”不看群体的模型 + reweighing”几乎没用。

(a) $P_w(a,y) = w(a,y)P(a,y)/Z$，其中 $Z = \sum_{a,y} w(a,y)P(a,y) = \sum_{a,y}P(a)P(y) = 1$，所以 $P_w(a, y) = P(a)P(y)$；平均权重就是 $Z = 1$。(b) 群体 $a$ 的截距只影响该群体的样本；加权 log loss 关于该群体预测概率 $q_a$ 的最优解是加权还款率 $\sum_{i \in a} w_i y_i / \sum_{i \in a} w_i = P_w(Y=1 \mid a) = P(Y=1)$（由 (a)，$Y$ 与 $A$ 在加权分布下独立）。两个群体的 $q_a$ 相同，所以无论阈值多少，批准率都相同。(c) 看群体的模型有一个”群体截距”，加权正好把它推到 (b) 的位置（再加上特征的作用）；不看群体的模型没有这个自由度，只能通过和群体相关的特征去近似这个平移，而那些特征的系数同时要服务于预测，所以只能挪一点点。

练习 2（协方差代理与 reductions）. (a) For binary $a$ with $P(a = 1) = \pi$, show that $\frac{1}{n}\sum_i (a_i - \bar a) z_i = \pi(1 - \pi)\,(\bar z_1 - \bar z_0)$, where $\bar z_a$ is the mean logit in group $a$. (b) Give a two-group example where the mean logits are equal but the approval rates at threshold 0 differ a lot. (c) In the reductions approach, the $Q$-player’s best response for fixed $\lambda$ is a cost-sensitive classification problem. For the DP constraint, what do the costs look like, and how is that related to reweighing?

(a) $\frac1n\sum_i (a_i - \bar a)z_i = \frac1n\sum_{i: a_i = 1}(1-\pi)z_i + \frac1n\sum_{i: a_i=0}(-\pi)z_i = \pi(1-\pi)\bar z_1 - (1-\pi)\pi\bar z_0$。所以协方差为 0 ⇔ 两个群体的平均 logit 相等。(b) group 0 的 logit 全是 0.1（全部批准），group 1 的 logit 一半是 +2.1、一半是 −1.9（平均也是 0.1，批准率 50%）。平均相等，批准率差 0.5 —— 代理只控制一阶矩。(c) Lagrangian 里 DP 约束是”每个群体的批准率减去总体批准率”的线性函数，乘以 $\lambda$ 后变成：把某个群体的人”预测为 1”额外加上（或减去）一个与群体有关的常数代价。这等价于给每个 (群体, 标签) 的样本换一个权重、必要时翻转标签 —— 是一种随训练动态调整的 reweighing。reductions 迭代地找这些权重，并输出一个随机化的混合分类器，所以它对任何能做加权训练的学习器都适用。

练习 3（Hardt 的线性规划）. 两个群体各占一半，base rate 都是 0.5。group 0 的 ROC 上凸包顶点是 (0, 0)、(0.1, 0.6)、(0.4, 0.9)、(1, 1)；group 1 的是 (0, 0)、(0.3, 0.6)、(1, 1)。(a) 写出期望错误率作为共同工作点 $(x, y)$ 的函数。(b) 求 equalized odds 下错误率最小的点。(c) 为两个群体分别写出实现这个点的随机化规则（用”在两个顶点之间混合”和”与对角线混合”描述）。

(a) $\text{err} = 0.5[0.5(1-y) + 0.5x] \times 2 = 0.5(1 - y + x)$，即最小化 $x - y$（等价于最大化 TPR − FPR）。(b) 交集是两条包络中较低者的下方。group 1 的包络：$x \in [0, 0.3]$ 上 $y = 2x$，$[0.3, 1]$ 上 $y = 0.6 + 0.4(x - 0.3)/0.7$。group 0 的包络在 $x \in [0, 0.1]$ 上 $y = 6x$，处处高于 group 1 的（检查顶点：在 0.1 处 0.6 > 0.2，在 0.3 处 0.8 > 0.6，在 0.4 处 0.9 > 0.657）。所以可行域就是 group 1 的包络下方，$y - x$ 在 group 1 的顶点 (0.3, 0.6) 取最大 0.3（另一段斜率 0.4/0.7 < 1，向右走 $y - x$ 减小）。最优点 (0.3, 0.6)，错误率 0.35。(c) group 1：恰好是一个顶点 → 一个确定性阈值。group 0：(0.3, 0.6) 在它的包络（该处高 0.8）下方。从对角线点 $D = (0.45, 0.45)$ 沿 (−1, +1) 方向走，经过目标点（$s_p = 0.15$ 处）到达包络上的 $C = (0.45 - s, 0.45 + s)$。包络在 $x \in [0.1, 0.4]$ 段是 $y = x + 0.5$，所以 $0.45 + s = 0.45 - s + 0.5$，$s_c = 0.25$，$C = (0.2, 0.7)$。$C$ 是顶点 (0.1, 0.6) 和 (0.4, 0.9) 的混合，后者的权重 $q = (0.2 - 0.1)/0.3 = 1/3$：两个阈值之间的人以 1/3 的概率批准。$\lambda = s_p / s_c = 0.6$：以 0.6 的概率用 $C$ 的规则，否则以概率 $r = 0.45$ 随机批准。检查：$0.6 \times (0.2, 0.7) + 0.4 \times (0.45, 0.45) = (0.3, 0.6)$。

练习 4（EO 下的批准率差距）. (a) 设两个群体的 TPR 都是 $t$、FPR 都是 $f$（equalized odds 精确成立）。证明批准率差距等于 $(p_0 - p_1)(t - f)$。(b) 用 §5.2 的 EO 目标点 (0.520, 0.884) 和测试集 base rate 差距 0.141 预测 Hardt EO 的批准率差距，和测试集上实测的 +0.047 比较。(c) 为什么说”EO 失败了，因为批准率还不相等”是一个误解？

(a) $P(\hat Y = 1 \mid A = a) = p_a t + (1 - p_a) f = f + p_a(t - f)$，相减即得。(b) $0.141 \times (0.884 - 0.520) = 0.141 \times 0.364 \approx 0.051$，与实测 +0.047 接近（测试集上的 TPR、FPR 和验证集上的目标点略有不同，TPR gap −0.000、FPR gap −0.016 也不是严格为 0）。(c) 只要 base rate 不同、分类器有用（$t > f$），EO 就要求批准率不同。批准率相等是 DP 的要求；两者在 $p_0 \ne p_1$ 时不能同时成立（第 5 讲 §4.3）。

练习 5（equalize 为什么让 WEAT 恰好为 0）. (a) 证明：hard debiasing 之后，对任何被中和的词 $w$（$w \cdot g = 0$）和任何被 equalize 的一对 $(a, b)$，有 $\cos(w, a) = \cos(w, b)$。(b) 由此说明：当 WEAT 的属性集 $A$、$B$ 恰好由 equalize 过的词对组成时，每个 $s(w)$ 都是 0。(c) 为什么用名字做属性集时 WEAT 几乎不变？要让它也变成 0，hard debiasing 需要怎么改？这样改会不会引入新的问题？

(a) equalize 后 $a = \nu + \beta u$、$b = \nu - \beta u$，其中 $u$ 是沿 $g$ 的单位向量（$\pm g$），$\nu \perp g$，两者长度都是 1。所以 $w \cdot a - w \cdot b = 2\beta\, w \cdot u = 0$（$w \perp g$）。(b) $s(w) = \text{mean}_A \cos(w, a) - \text{mean}_B \cos(w, b)$，若 $A$、$B$ 按对排列，逐对相减都是 0。(c) 名字没有被 equalize，它们在 $g$ 之外的方向上也不同（例如 John 和 career 词的共现），中和 career 词只去掉了 $g$ 上的分量。把名字也加进 equalize 词对可以让这个 WEAT 也变成 0 —— 但那仍然是”测量用的词表 = 去偏用的词表”，WEAT 归零不能再证明什么；而且偏见可能转移到下一组没被列入的词（语境词、刻板化描述词）上，§7.5 的探针仍然能读出来。

练习 6（写一个缓解方案）. 你为一家贷款机构工作。法律要求：决策时不能使用种族；但在模型开发和公平审计中可以使用（经过隐私审批的）种族数据。审计发现：模型对某少数群体的批准率低 0.2，真实还款数据显示该群体的 TPR 也低 0.15；另外有证据表明该群体过去的”违约”记录中有一部分是催收政策造成的。写一页方案：(a) 用 §1.2 的方法诊断差距来源需要哪些分析；(b) 你会选哪一类方法，为什么排除其他几类；(c) 你要报告哪些指标、在什么数据上、怎样给出不确定性；(d) 你预期的代价是什么，要向谁解释。

(a) 分段测量：两个群体真实还款率的差距（世界）；用群体感知的模型做一次仅用于诊断的拟合，看差距里有多少来自 aggregation（第 1 讲的 thin-file 那一半）；检查标签：被怀疑误记的记录占多少、和哪些特征相关（第 5 讲 §7.4：误记的结构决定哪个指标被骗）。(b) 决策时不能用种族 → 排除按群体阈值、Hardt EO 和 DI remover 的按群体修复。标签可疑 → 对完全依赖标签的约束（EO）保持警惕，优先修标签（例如把政策导致的违约重新标注或剔除，再用 reweighing 平衡）；然后用只在训练时用群体的方法（协方差约束、reductions 或对抗训练），目标选 equal opportunity（“会还款的人被拒”是主要伤害），并像 §4.1 那样检查系数在群体内部造成的扭曲。(c) 在独立的审计集上报告准确率、批准率、TPR、FPR、PPV 的差距和群体内校准，全部带 bootstrap 置信区间；交叉群体（第 5 讲 §9.2）；分别用”记录的标签”和”修正后的标签”计算，并说明两者的差别。(d) 预期准确率下降约几个百分点、被批准者的违约率在少数群体中上升（PPV 差距扩大）；需要向风控（违约成本）、合规（为什么训练时使用了种族数据、为什么不构成 disparate treatment）和受影响的申请人（拒绝理由）解释。写进 model card，并承诺部署后按季度重新测量。

代码相关的练习（“动手改一改”）见 code/lecture06/README.md。

12. 延伸阅读 · Further reading

本讲的主要来源

• CS329T Spring 2022 第 5 周 slides “Fairness in ML”（2022-04-26，Datta & Mitchell）与课程概览；HW3（Spring 2021 版本：adversarial debiasing 与 word2vec 去偏）；Lab 4。
• Zhang, Lemoine & Mitchell, “Mitigating Unwanted Biases with Adversarial Learning”, AIES 2018.
• Bolukbasi, Chang, Zou, Saligrama & Kalai, “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings”, NeurIPS 2016.
• Caliskan, Bryson & Narayanan, “Semantics derived automatically from language corpora contain human-like biases”, Science 2017.
• Leino, Black, Fredrikson, Sen & Datta, “Feature-Wise Bias Amplification”, ICLR 2019.
• Lu, Mardziel, Wu, Amancharla & Datta, “Gender Bias in Neural Natural Language Processing”, 2018（arXiv；后收入 2020 年的文集）；Sun et al., “Mitigating Gender Bias in Natural Language Processing: Literature Review”, ACL 2019.
• Leslie, “Understanding bias in facial recognition technologies: an explainer”, The Alan Turing Institute, 2020.

按干预点

• Pre-processing：Kamiran & Calders, “Data preprocessing techniques for classification without discrimination”, KAIS 2012；Feldman, Friedler, Moeller, Scheidegger & Venkatasubramanian, “Certifying and Removing Disparate Impact”, KDD 2015；Zemel, Wu, Swersky, Pitassi & Dwork, “Learning Fair Representations”, ICML 2013；Madras, Creager, Pitassi & Zemel, “Learning Adversarially Fair and Transferable Representations”, ICML 2018.
• In-processing：Zafar, Valera, Gomez Rodriguez & Gummadi, “Fairness Constraints: Mechanisms for Fair Classification”, AISTATS 2017；Agarwal, Beygelzimer, Dudík, Langford & Wallach, “A Reductions Approach to Fair Classification”, ICML 2018.
• Post-processing：Hardt, Price & Srebro, “Equality of Opportunity in Supervised Learning”, NeurIPS 2016；Pleiss, Raghavan, Wu, Kleinberg & Weinberger, “On Fairness and Calibration”, NeurIPS 2017；Woodworth, Gunasekar, Ohannessian & Srebro, “Learning Non-Discriminatory Predictors”, COLT 2017.
• 批评与权衡：Lipton, McAuley & Chouldechova, “Does mitigating ML’s impact disparity require treatment disparity?”, NeurIPS 2018；Suresh & Guttag, “A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle”, EAAMO 2021；Friedler, Scheidegger & Venkatasubramanian, “On the (im)possibility of fairness”, 2016.
• Bias amplification：Zhao, Wang, Yatskar, Ordonez & Chang, “Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints”, EMNLP 2017.

NLP 与 LLM

• Gonen & Goldberg, “Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them”, NAACL 2019；Nissim, van Noord & van der Goot, “Fair is Better than Sensational: Man is to Doctor as Woman is to Doctor”, Computational Linguistics 2020；Ethayarajh, Duvenaud & Hirst, “Understanding Undesirable Word Embedding Associations”, ACL 2019；Goldfarb-Tarrant et al., “Intrinsic Bias Metrics Do Not Correlate with Application Bias”, ACL 2021.
• Zhao et al., “Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods”（WinoBias）, NAACL 2018；Rudinger et al., “Gender Bias in Coreference Resolution”（Winogender）, NAACL 2018；Maudslay et al., “It’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution”, EMNLP 2019.
• Parrish et al., “BBQ: A Hand-Built Bias Benchmark for Question Answering”, Findings of ACL 2022；Blodgett et al., “Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets”, ACL 2021；Ganguli et al., “The Capacity for Moral Self-Correction in Large Language Models”, 2023.
• Levy & Goldberg, “Neural Word Embedding as Implicit Matrix Factorization”, NeurIPS 2014；Levy, Goldberg & Dagan, “Improving Distributional Similarity with Lessons Learned from Word Embeddings”, TACL 2015（PPMI + SVD 与 word2vec 的关系）。

工具：IBM AIF360（实现了 reweighing、disparate impact remover、adversarial debiasing、equalized odds 后处理等）；Microsoft Fairlearn（reductions、threshold optimizer）。读它们的文档时，留意每个方法”需要什么”（§2.2）。

13. 下一讲预告 · Next lecture

Lecture 7：对抗样本与攻击（Adversarial Examples & Attacks）。

本讲的对手是”帮忙”的：它试图从预测里猜出群体，我们训练模型让它猜不出。第 7 讲的对手是”捣乱”的：它在输入上加一个人眼几乎看不见的扰动，让模型把熊猫认成长臂猿。两者共享同一套数学 —— 对一个可微的模型，对输入（或参数）求梯度、朝”让某个 loss 变大”的方向走一步。第 1 讲 §5.3 已经看到，在线性模型上最坏情况的扰动比随机扰动强几百倍；第 7 讲把它推广到深度网络：threat model（白盒 / 黑盒、$L_p$ 球、物理世界）、FGSM 与 PGD、C&W、迁移攻击，以及 “adversarial examples are not bugs, they are features” —— 模型依赖的那些对人类不可见、却真实存在于数据里的”非鲁棒特征”。你会发现它和本讲 §7.5 的教训惊人地相似：模型依赖的信息，不一定在你以为的那个方向上。