Lecture 2 · 解释与 Shapley 值 · Explanations & Shapley Values

对应 Maps to：Spring 2022 Week 2 —— Anupam Datta & John Mitchell 的两次 “Explanations” 讨论课（Apr 5、Apr 12, 2022：explanation taxonomy、local explanations、input importance and Shapley values、“Shapley values in explanations: SHAP & QII”、LIME），阅读材料 Ribeiro, Singh & Guestrin 2016（LIME, “Why Should I Trust You?”）、Datta, Sen & Zick 2016（Quantitative Input Influence）、Lundberg & Lee 2017（SHAP）、Lage et al. 2019（“An Evaluation of the Human-Interpretability of Explanation”）；Homework 2 Part I（LIME、Shapley / QII / SHAP、TreeSHAP）与 Lab 2；Fall 2023 Lecture 12 “Interpretability”（Datta、Mitchell、Taly；slides 标题页写作 Lecture 11）中关于 ablation、what-if 与公理化的部分，以及 Taly 的相关论文 Merrick & Taly 2020（“The Explanation Game”）、Sundararajan & Najmi 2020（“The Many Shapley Values for Model Explanation”）。 配套代码 Code：code/lecture02/（numpy；5 个脚本 + 2 个工具模块，每个脚本 20 秒以内） 预计阅读时间：3 小时（含运行代码）

0. 本讲导航 · How to read this lecture

0.1 Learning goals

After this lecture you should be able to:

1. 用四个问题给任何一种解释方法定位：解释哪个输出（概率、logit、决策、loss、群体统计量；局部还是全局）、用哪些输入、需要怎样的模型访问（model-agnostic vs model-specific）、在哪个阶段（post hoc vs intrinsic）。
2. Explain with concrete toy functions (AND, OR, XOR) why one-at-a-time ablation gives contradictory answers whenever features interact or are redundant, and why the fix is to average marginal contributions over all contexts.
3. 写出 Shapley 值的子集形式和排列形式，说出四条公理（efficiency、symmetry、dummy、additivity）各自的含义，并能用 unanimity games / Harsanyi 分红讲清楚为什么这四条公理唯一确定了 Shapley 值。
4. Compute Shapley values exactly (2^d), by permutation sampling with error bars, and by KernelSHAP (weighted least squares with the Shapley kernel); know what TreeSHAP exploits; and recognize that the background sample is a second source of error.
5. 解释 LIME 的目标函数和核宽度的作用；说明 LIME 和 KernelSHAP 是”同一个回归、不同的权重”；分清斜率（敏感度）和归因（贡献）。
6. 对”把一个特征去掉是什么意思”给出至少四种回答（单一基线、interventional、conditional、contrastive 参考分布），说出每一种的代价：dummy 被打破、off-manifold 查询、估计困难；并能为一个具体场景（比如信贷拒绝理由）选择并论证参考分布。

0.2 先修与本讲范围 Prerequisites & scope

本讲只讨论表格数据上的特征归因（feature attribution）。同一套思想（“把输出分配给输入”）在第 3 讲搬到图像和文本上（Integrated Gradients 就是 Shapley 值的连续版本 Aumann–Shapley），在第 4 讲搬到训练样本上。

0.3 本讲地图 Map

§1  解释的分类：解释什么输出、用什么输入、怎样访问模型、什么阶段          ← 代码 05 演示 2
§2  "一次改一个特征"为什么不够：AND / OR / XOR                        ← 代码 01 演示 1
§3  QII：干预、影响、集合影响；票数 ≠ 权力                             ← 代码 01 演示 2；代码 05 演示 3
§4  合作博弈与 Shapley 值：定义、四条公理、唯一性、交互、线性模型         ← 代码 01 演示 3–5
§5  怎样算：精确枚举、排列采样、KernelSHAP、TreeSHAP、背景样本的误差      ← 代码 02
§6  LIME：局部代理模型、核宽度、和 KernelSHAP 的关系、斜率 ≠ 归因         ← 代码 03
§7  "把一个特征去掉"是什么意思：基线、interventional vs conditional、
    off-manifold、The Explanation Game                                ← 代码 04；代码 02 演示 4
§8  从局部到全局；信贷案例：找回泄露、群体差距、拒绝理由                  ← 代码 05
§9  局限：解释不是因果、人类可理解性、可被操纵
§10–14 小结、误区、练习、延伸阅读、下一讲预告

0.4 Notation

0.5 关于数字 · About the numbers

讲义里的所有实验数字都来自配套代码的实际输出（固定随机种子，同一台机器上重复运行逐字相同）。数据是合成的：第 1 讲的信贷世界（trust_data.py），加上本讲的扩展版（credit_models.py：4 个”和真正原因相关、但本身不影响还款”的字段）。因为世界是我们造的，所以我们知道哪个字段是泄露、哪个字段模型根本没读、真实的 thin-file 效应有多大 —— 这些”标准答案”让我们能检验解释方法，而不只是看它们”像不像”。

1. 解释的分类 · A taxonomy of explanations

1.1 为什么要解释 · Why explain?

第 1 讲的排行榜冠军 Model A 在测试集上准确率 0.912，部署时跌到 0.710；一行 permutation importance 就把泄露字段 officer_flag 找了出来。这是解释方法最朴素、也最重要的用途：debugging。CS329T 2022 年的 explanations 模块开宗明义：“Explanations enable understanding and debugging of ML models”；2023 年 Taly 等人的 interpretability 讲座列出了四个用途：debugging model predictions、generating an explanation for the end-user、analyzing model robustness、monitoring models in production。

2023 年讲座里的例子值得复述。一个视觉问答（VQA）模型在精心构建的数据集上测试准确率 61%；问它 “How symmetric are the white bricks on either side of the building?”，它答 “very”（正确）。但把问题改成 “how asymmetric…”、“how soon are the bricks fading…”，甚至 “how fast are the bricks speaking…”，它还是答 “very”。归因方法显示，“symmetrical” 这个词几乎没有得到任何 attribution —— 模型只看问题的类型（“how …”）来选答案（Mudrakarta et al. 2018, “Did the model understand the question?”）。Right for the wrong reason, invisible to test accuracy, visible to an attribution.

1.2 Four questions that place any explanation method

The 2022 lectures organized explanation methods by four questions about the pipeline inputs → model → outputs:

1. Explanation of what output? Scope: a single prediction (local) or the model’s behavior on a population (global). Type: the predicted probability, the logit, the classification outcome, the loss, or an aggregate quantity such as an approval-rate disparity between groups.
2. Explanation in terms of what input? Raw features (this lecture), internal neurons (Lecture 3), or training examples (Lecture 4).
3. With what model access? Model-agnostic methods only query $f$ as a black box (LIME, QII, KernelSHAP, permutation importance). Model-specific methods exploit structure — gradients (Integrated Gradients, Lecture 3) or tree paths (TreeSHAP).
4. At what stage? Intrinsically interpretable models (a short scorecard, a GAM, a small tree) versus post hoc explanations of an already-trained model. Everything in this lecture is post hoc.

第一个问题最容易被忽略。代码 05 演示 2 对第 1 讲的第 31 号申请人分别解释树模型 A 的 logit、概率和 0/1 决策：三组数字不同，例如 history 和 officer_flag 的贡献之比在 logit 上是 0.13、在概率上 0.12、在决策上 0.21。对这个人排序恰好没变；在 100 个被拒申请人上，logit 和概率的首要理由 98% 一致，但换成 0/1 决策时有 6–8% 的人首要理由变了。说”首要理由是 X”之前，先说清楚解释的是哪个量。

1.3 一个统一的框架：查询、比较、汇总 · Query, compare, summarize

Datta 的 QII 讲稿把所有”输入重要性”方法拆成三步：

1. Query definition：我们到底想解释什么？“为什么 Jane 的分数是 665？""为什么模型拒绝了 Jane？""为什么模型对两个群体有 disparate impact？”
2. Output comparison：拿什么和什么比？—— 改变某些输入（干预），比较前后的输出（“causal testing”）。
3. Summarization：把比较的结果汇总成每个特征一个数（“665 里，DTI 贡献了 133，income 贡献了 −45……”）。

Covert, Lundberg & Lee (2021, “Explaining by Removing”) made the same observation formally and showed that a large family of methods — LIME, SHAP, QII, permutation importance, occlusion, and more — are removal-based explanations that differ in exactly three choices:

本讲的结构就是沿着这张表走：§2–§4 讲第三列的”汇总”（为什么是 Shapley），§5–§6 讲怎样算、LIME 怎样汇总，§7 讲第一列的”去掉”——它看起来最技术，实际上决定一切。

1.4 What do we want from an explanation?

The 2022 slides listed desiderata: local agreement with the model’s predictions; not too local (consistency between local and global explanations); causal (with respect to the model: changing an input the explanation calls important should change the output); consistency / monotonicity (if a model changes so that a feature matters more, its attribution should not go down); completeness and dummy. Another slide phrased the same goals as “answer a rich set of queries, capture causal influence, reflect the power of a feature, be accurate”.

另一个区分会贯穿本讲：faithfulness（解释忠实于模型实际的计算）vs plausibility（解释看起来合理、符合人的直觉）。2023 年讲座的说法是：“The mandate for attributions is to be faithful to the model’s reasoning.” 一个讨人喜欢但不忠实的解释，会提高信任、同时降低可信度（第 1 讲 §4.1）。

2. “一次改一个特征”为什么不够 · Why one-at-a-time fails

2.1 两种最朴素的做法

给定被解释的点 $x$ 和一个”基线” $b$（暂时把它当作”特征不在场时的值”，§7 再追问它是什么），定义 $v(S) = f(x_S, b_{\bar S})$。最自然的两种归因：

• Leave-one-out（ablation / occlusion）：$\phi_i^{\text{LOO}} = v(N) - v(N \setminus \{i\})$ —— 其他特征都在场时，去掉 $i$ 损失多少。
• Add-one-in：$\phi_i^{\text{add}} = v(\{i\}) - v(\emptyset)$ —— 其他特征都不在场时，只加 $i$ 得到多少。

2023 年的讲座把 ablation 列为 “naive approach”，指出它的三个问题：computationally expensive、unrealistic inputs（§7.3）、misleading when features interact。第三个问题用四个玩具函数就能看清（代码 01 演示 1；$x$ = 全 1，$b$ = 全 0）：

2.2 读这张表

• AND（互补）：去掉任何一个，输出都从 1 掉到 0，所以 LOO 让每个特征”负全责”，加起来多算了一倍；只加一个时输出都还是 0，add-one-in 说”谁都没贡献”。两种”一次一个”的方法互相矛盾。
• OR（冗余）：两个特征互为备份，去掉任何一个都没关系 —— LOO 说”模型对每个特征都不敏感”，可是输出明明来自它们。冗余在真实数据里无处不在：收入和存款、负债率和信用卡使用率、同一个信息的两种编码。
• 信贷规则：LOO 说”只有收入重要”。如果用它写批准 / 拒绝理由，就会漏掉”没有逾期”这个真实起作用的条件（它和”信用历史长”互为备份）。

共同的原因：特征 $i$ 的边际贡献 $v(S \cup \{i\}) - v(S)$ 取决于情境 $S$（其他特征在不在场）。“一次改一个”只看了一个情境（$S$ = 其他所有人，或 $S = \emptyset$）。自然的修正是：对所有情境取平均。Banzhaf 值给每个情境相同的权重；Shapley 值给每个大小的情境相同的总权重（§4.1）。两者都给 AND 和 OR 一个对称、合理的答案；只有 Shapley 的总和一定等于 $f(x) - f(b)$（信贷规则上 Banzhaf 的和是 1.25）。

2.3 什么时候”一次一个”就够了

如果模型是可加的（$f(x) = \sum_j g_j(x_j)$），每个特征的边际贡献在所有情境下都一样，LOO、add-one-in、Banzhaf、Shapley 全部相等。第 1 讲的线性模型在 logit 上就是可加的 —— 所以 θ_j·x̃_j 是一个无可争议的解释。但在概率上它就不是了：代码 01 演示 5 对第 1 讲第 31 号申请人（线性 Model B），概率空间里 LOO 的总和是 0.303，而要分配的总量只有 0.189；sigmoid 把可加的 logit 压扁，制造了交互（最大的交互指数约 0.056，logit 上是 1e-16）。非线性模型 —— 树、神经网络 —— 到处都是交互。

3. QII：Quantitative Input Influence

3.1 干预与影响

Datta, Sen & Zick（IEEE S&P 2016）是 Shapley 值进入模型解释的早期工作之一（在 Lundberg & Lee 2017 之前），标题里的 “algorithmic transparency” 说明了它的动机：给受算法决策影响的人一份 “transparency report”。（论文还证明了 QII 的敏感度很低，只需加很少的噪声就能让 transparency report 满足 differential privacy —— 解释和隐私的联系，第 10 讲。）QII 有三个要素：

• Quantity of interest（QoI） $Q$：想解释的量。可以是一个人的结果（“Jane 被批准的概率”），也可以是一个群体的量（“女性的批准率”），甚至是群体之间的差距（“两个群体批准率之差”——disparate impact）。
• Intervention：把特征集合 $S$ 换成随机值 $U_S$，切断它们和这个人 / 这群人的联系，记作 $X_{-S}U_S$。论文的定义是：从数据分布里独立地再抽一整个人 $u$，$X_{-S}U_S$ 在 $S$ 上取 $u$ 的值、在其余特征上保留原值（$S$ 内部的相关性保留，$S$ 和其余特征之间的相关性被切断）。Merrick & Taly（2020）把 QII 描述成另一种做法：$S$ 里每个特征各自独立地从自己的边缘分布抽（“joint-marginal”）。特征相互独立时两者相同；代码里的 qii_set_game 两种都实现了，默认用论文的定义。
• Influence：干预前后 $Q$ 的变化：

$|S| = 1$ 时叫 unary QII。第 1 讲的 permutation importance 正是 $Q$ = 准确率的 unary QII（打乱一列 = 从边缘分布重抽这一列）。对一个人的预测，unary QII 就是 interventional 版的 leave-one-out —— 所以它有 §2 的全部问题。

3.2 从集合影响到 Shapley 值

QII 的解决办法和 §2 一样：先定义集合影响 $\iota(S)$，再看特征 $i$ 的边际影响 $\iota(S \cup \{i\}) - \iota(S)$，最后”averaged over all subsets of features that it could combine with”（2022 年讲稿原话的意思）。用 Shapley 值做这个平均，就是 Shapley-QII；论文的附录还讨论了 Banzhaf 指数和 Deegan–Packel 指数这两种汇总方式。

一个细节：QII 的博弈是”去掉 $S$ 损失多少”，SHAP 的博弈是”保留 $S$ 得到多少”。两者互为对偶：$\iota(S) = v(N) - v(N \setminus S)$。对偶博弈的 Shapley 值和原博弈完全相同（代码 01 演示 3 数值验证：差 5.6e-17）。所以只要参考分布相同，“Shapley-QII”和”interventional SHAP”是同一个数 —— 后来的文献之所以看起来五花八门，差别几乎全在”参考分布”上（§7.4）。

3.3 票数 ≠ 权力 · Which states decide the winner?

2022 年的讲稿用美国大选做类比：问”哪些州贡献的选举人票最多？“和问”哪些州决定了胜负？“是两个不同的问题。后者是权力（power），合作博弈论里叫 Shapley–Shubik power index（Shapley & Shubik 1954）：一个玩家的权力 = 随机排队时，TA 恰好是”把联盟从输变成赢”的那个人的概率。代码 01 演示 2：

49 票和 1 票的权力一样（第一行）；有 3 票的 P4 没有任何权力（第三行）—— 它是一个 dummy player：加入任何联盟都不改变结果。映射到模型：一个特征的”大小”（权重、取值、在训练数据里的方差）不等于它对这个决策的影响。另一个细节：Banzhaf 的原始值加起来不等于 1，需要手动归一化 —— 它不满足 efficiency。

QII 的另一个独特之处是 QoI 可以是群体统计量。2022 年的讲稿比较各方法能回答的”查询”：解释个体预测，QII、permutation importance、LIME、KernelSHAP、TreeSHAP 都可以；解释 “disparate impact ratio” 这样的群体性质，只有 QII 可以。§8.2 用它分解第 1 讲里 Model B 的群体批准率差距。

4. 合作博弈与 Shapley 值 · Cooperative games and the Shapley value

4.1 Definition

A cooperative game is a set of players $N$ and a characteristic function $v: 2^N \to \mathbb R$ with $v(S)$ the payoff coalition $S$ can secure. Lloyd Shapley (1953) asked: if all players cooperate and earn $v(N)$, how should the surplus $v(N) - v(\emptyset)$ be divided? His answer:

The weights look arbitrary until you read them as a probability. Line the players up in a uniformly random order and let each one join the coalition of everyone ahead of them. Player $i$ finds exactly the set $S$ ahead with probability $|S|!\,(d - |S| - 1)!/d!$. So the permutation form is

The Lab 2 slides gave the same picture: “The feature values enter a room in random order… The Shapley value of a feature value is the average change in the prediction that the coalition already in the room receives when the feature value joins them.” Another way to see the weights: every coalition size $s = 0, \dots, d-1$ receives total weight $1/d$, split equally among the $\binom{d-1}{s}$ coalitions of that size. Banzhaf instead gives every coalition the same weight $1/2^{d-1}$, so mid-sized coalitions dominate.

For explanations, the players are features and $v(S)$ is the model’s output “when only the features in $S$ are known”. With that game, $\sum_i \phi_i = v(N) - v(\emptyset) = f(x) - \text{(baseline output)}$.

4.2 四条公理 · The four axioms

Theorem (Shapley 1953). The Shapley value is the only attribution rule satisfying all four axioms.

4.3 Why the axioms force uniqueness · 唯一性证明的骨架

证明的思路值得一步一步看，因为它告诉你每条公理在干什么。

Step 1: unanimity games. For a nonempty $T \subseteq N$ define $u_T(S) = 1$ if $T \subseteq S$ and 0 otherwise — “payoff 1 only when all of $T$ are present”. In $u_T$, players outside $T$ are dummies, so dummy gives them 0. Players inside $T$ are pairwise symmetric, so symmetry gives them equal shares. Efficiency says the shares sum to $u_T(N) - u_T(\emptyset) = 1$. Hence $\phi_i(c\,u_T) = c/|T|$ for $i \in T$ and 0 otherwise — the axioms leave no freedom at all on these games. (Scaling by $c$ works the same way.)

Step 2: every game is a combination of unanimity games. The $2^d - 1$ games $u_T$ form a basis of all games with $v(\emptyset) = 0$:

The coefficients $m(T)$ are the Harsanyi dividends: the part of the payoff produced by exactly the coalition $T$ together, not attributable to any smaller sub-coalition — a pure $|T|$-th order interaction.

Step 3: additivity glues the pieces. $\phi_i(v) = \sum_{T \ni i} m(T)/|T|$: every coalition’s dividend is split equally among its members. Any rule satisfying the four axioms must produce this number, and the Shapley formula does. (Dividends can be negative; Step 1 never used the sign of $c$, so nothing changes.)

代码 01 演示 3(b) 把信贷规则 income ∧ (no_missed ∨ long_history) 分解成分红：

两个”二人联盟”各产生 1；三人联盟有一个 −1 的分红（两个备份条件同时满足时，第二个是多余的）。income 得 1/2 + 1/2 − 1/3 = 2/3，另外两个各得 1/2 − 1/3 = 1/6 —— 正是 §2 表里的 Shapley 值。

每条公理都是必要的. 代码 01 演示 3(c) 在随机博弈上数值检查六种候选方法：

最后一行很有启发：用”缩放”来修复 efficiency 看似自然，但缩放因子依赖于博弈本身，于是破坏了 additivity。演示 3(a) 还验证了同一个 Shapley 值的四条计算路径 —— 子集公式、枚举 120 个排列、分红平分、KernelSHAP 的加权最小二乘（§5.3）—— 结果相差不超过 3e-16。

4.4 Other axiomatizations

• Young (1985) replaced additivity and dummy by a single strong monotonicity axiom: if a player’s marginal contributions in game $w$ are at least those in game $v$ for every coalition, its value in $w$ is at least its value in $v$. Efficiency + symmetry + strong monotonicity again characterize Shapley. The 2022 slides listed exactly “symmetry, dummy, monotonicity”, and stressed that monotonicity is “necessary to compare feature influence scores of individuals” — without it, a feature could matter more for Alice’s prediction yet receive less credit. The QII paper justifies its choice of the Shapley value with exactly this triple (symmetry, efficiency, monotonicity).
• Lundberg & Lee (2017) framed explanations as additive feature attribution models $g(z') = \phi_0 + \sum_i \phi_i z'_i$ over binary “present / absent” variables $z'$, and proved (Theorem 1, building on Young) that local accuracy, missingness and consistency single out the Shapley values. They called the resulting values SHAP (SHapley Additive exPlanations).

So “SHAP”, “Shapley-QII” and “the Shapley value of the explanation game” are the same solution concept. What differs between implementations is the game — Sundararajan & Najmi’s (2020) title says it best: the many Shapley values for model explanation.

4.5 交互 · Interactions

Shapley 值是一个一阶摘要：它把每个分红平分给成员，分配本身是公平的，但交互的结构在平分之后就看不见了。最极端的例子是 XOR：$f(x) = f(b) = 0$，Shapley 值 (0, 0)，可是这个函数的全部行为都来自交互。

Shapley interaction index（Grabisch & Roubens 1999）用二阶差分代替一阶差分：

代码 01 演示 4：AND 的 $I_{12} = +1$（互补），OR 是 −1（替代 / 冗余），XOR 是 −2；信贷规则里 income 和另外两个条件各 +0.5，no_missed 和 long_history 之间 −0.5（有一个就够）。Lundberg et al.（2020）的 SHAP interaction values 是同一个思路（把 $I_{ij}$ 平分到 $(i, j)$ 和 $(j, i)$ 两个格子里）。实践中，“一阶归因 + 最大的几个二阶交互”往往比单独的一阶归因诚实得多。

4.6 线性模型：兑现第 1 讲的承诺

第 1 讲说：“对线性模型、以平均值为基线时，Shapley 值恰好就是 $\theta_j \tilde x_j$。“现在可以证明：对 $f(x) = \theta_0 + \sum_j \theta_j x_j$ 和 interventional 博弈 $v(S) = \mathbb E_{R}[f(x_S, R_{\bar S})] = \theta_0 + \sum_{j \in S}\theta_j x_j + \sum_{j \notin S}\theta_j \mathbb E[R_j]$，特征 $j$ 在任何情境下的边际贡献都是 $\theta_j(x_j - \mathbb E[R_j])$，所以

代码 01 演示 5 用第 1 讲的 Model B 和第 31 号申请人验证（背景 = 训练集 4000 人）：

logit 上逐位相同（差 2e-16）。概率上 Shapley 仍然满足 efficiency（0.1885 = 0.8642 − 0.6757），LOO 则多算了 60%。一个警告：这个干净的结论依赖于”线性 + interventional”。§7.2 会看到，同一个线性模型在 conditional 博弈下的 Shapley 值不再是 $\theta_j(x_j - \mathbb E X_j)$。

5. 怎样算 · Computing Shapley values

5.1 指数代价

精确计算需要全部 $2^d$ 个 $v(S)$；而每个 $v(S)$ 本身通常是对 $m$ 个背景点的平均（§7），即 $m$ 次模型调用。代码 02 演示 1：

在一般情况下，计算 Shapley 值是 #P-hard 的（Deng & Papadimitriou 1994 对一类图博弈证明了这一点；对模型解释，Van den Broeck et al. 2021 证明了它和”计算模型在特征分布下的期望输出”一样难）。所以只有两条路：采样近似（§5.2–5.3），或利用模型结构（§5.4）。

5.2 Permutation sampling, with error bars

The permutation form is an expectation, so estimate it by Monte Carlo (Castro et al. 2009; Štrumbelj & Kononenko 2010, 2014): draw $K$ random orderings, walk along each one adding features, and average each feature’s marginal contributions. Each ordering costs $d - 1$ new value evaluations (with $v(\emptyset)$ and $v(N)$ cached). Three properties make this the workhorse estimator:

• Unbiased, with a free standard error. The per-permutation contributions are i.i.d., so $\widehat{\text{SE}}_i = \text{sd}_i/\sqrt K$ and a 95% interval is $\hat\phi_i \pm 1.96\,\widehat{\text{SE}}_i$.
• Exact efficiency per sample. Along one ordering the marginal contributions telescope to $v(N) - v(\emptyset)$, so every estimate sums to the right total — all of the error is in how the total is split.
• Cheap variance reduction. Antithetic sampling pairs each ordering with its reverse: a feature that joins early in $\pi$ joins late in $\pi^R$, the two samples are negatively correlated, and their average has lower variance (Mitchell et al. 2022 study this and other quasi-Monte Carlo schemes).

代码 02 演示 3：9 特征的梯度提升树 T9，20 个申请人 × 20 个种子，误差（RMSE，括号里是相对于平均 |φ| = 0.041 的比例）：

• 误差大致按 $1/\sqrt{\text{预算}}$ 下降：预算 32 倍，普通排列采样的误差降到约 1/6（$\sqrt{32} \approx 5.7$）。
• 方差缩减很值：预算 64 时对偶排列和成对 KernelSHAP 的误差约是普通方法的 1/3 —— 相当于省下约 10 倍的预算。
• 误差棒在排列很少时覆盖不足：3 个排列时 95% 区间的实际覆盖率只有 0.79，7 个时 0.89，15 个以上才到 0.93–0.95（标准误本身是从几个样本估出来的）。
• 对拒绝理由更要紧的是排序：“最重要的特征”在预算 32 的普通采样里有 7–8% 的实验认错了，方差缩减后降到 1–3%。同一个申请人，换一个随机种子就换一个”主要理由”—— 在信贷场景里这是不能接受的。

注意 d = 9 时精确计算只要 512 次评估：预算 ≥ 512 时直接精确计算更好。采样的价值在 d 大的时候。

5.3 KernelSHAP: Shapley values as weighted least squares

Lundberg & Lee’s key computational insight: fit a linear model in the binary “present / absent” space, exactly as LIME does (§6), but with a special weighting. Solve

Theorem (Charnes et al. 1988; Lundberg & Lee 2017, Theorem 2): with this Shapley kernel, the solution is exactly the Shapley value. Code 01 demo 3(a) checks it on a random 5-player game: the difference is 1.4e-16. Three remarks on why this kernel:

• The two infinite weights are the constraints. $\pi(\emptyset)$ and $\pi(N)$ are infinite, which forces the fitted model through $v(\emptyset)$ and $v(N)$ — that is efficiency.
• Extreme coalitions get the most weight. A coalition of size 1 isolates a feature’s effect alone; a coalition of size $d - 1$ isolates its effect given everything else. Mid-sized coalitions are numerous but each says less about any single feature. The kernel is exactly the reweighting that makes the regression coefficients average the marginal contributions with Shapley’s weights. (Exercise 5 verifies this by hand for $d = 2$.)
• Sampling. In practice, draw coalitions from the kernel (first a size $s$ with probability $\propto (d-1)/(s(d-s))$, then a uniform subset of that size) and solve an unweighted constrained least squares. Paired sampling — always include the complement $N \setminus S$ too — reduces variance substantially (Covert & Lee 2021, “Improving KernelSHAP”). In the table above, paired KernelSHAP was the most accurate method at every budget from 64 up (at 32, antithetic permutations were marginally better); unpaired KernelSHAP was no better than plain permutation sampling.

KernelSHAP has no built-in error bar (Covert & Lee 2021 derive one); permutation sampling does. The two are complementary.

5.4 TreeSHAP: exploiting model structure (idea only)

For trees there is a completely different route. TreeSHAP (Lundberg, Erion & Lee 2018; Lundberg et al. 2020, Nature Machine Intelligence) computes Shapley values exactly in polynomial time. The idea:

1. A tree’s output depends only on which leaf an input lands in. For a subset $S$, “the expected output when only $S$ is known” can be computed recursively down the tree: at a node that splits on a feature in $S$, follow $x$‘s branch; at a node that splits on a feature outside $S$, take a weighted average of both branches.
2. The trick is to avoid enumerating the $2^d$ subsets. A single top-down pass tracks, simultaneously for all subset sizes, how much weight each path carries. The cost per tree is $O(LD^2)$ for $L$ leaves and depth $D$ instead of exponential in $d$.
3. There are two ways to weight “both branches”. The path-dependent variant uses the fraction of training samples that went each way (the node’s “cover”), which approximates a conditional expectation. The interventional variant uses a background dataset and matches §7’s interventional game, at a cost linear in the background size.

HW2 asks whether a feature with no influence on the prediction can receive a nonzero TreeSHAP attribution. In §7’s language: if the value function is a true conditional expectation, yes — correlated features share credit (§7.2). The path-dependent algorithm only follows splits that actually occur in the tree, so a feature never used for splitting gets exactly 0; the algorithm is an approximation of the conditional game, not the game itself. The question shows that one name (“SHAP”) can stand for different games.

The flip side of model structure: trees are piecewise constant, so their gradients are zero almost everywhere and the gradient methods of Lecture 3 are useless on them, while TreeSHAP is useless for neural networks.

5.5 第二个误差来源：背景样本

前面的”精确”值，都是对这 100 个背景点精确。背景是从总体里抽的，所以 $v(S)$ 本身就是估计。代码 02 演示 4 用 1000 个背景点的结果作为参考答案，从中反复抽 $m$ 个当背景、精确枚举：

这个结果超出了我写代码前的预想：m = 100 时，背景误差（16%）已经大于预算 64 的成对 KernelSHAP 的采样误差（3%），而”最重要的特征”只有 74% 的时候和参考答案一致。原因是这几个申请人的前几名特征贡献很接近。教训：报告 SHAP 值时要报告背景是什么、有多大，并检查结论对背景重抽样是否稳定。置信区间可以几乎免费地得到 —— §7.4 解释为什么（m = 400 时覆盖率 0.99 偏保守，是因为我们从有限的 1000 人里不放回地抽了 40%）。

6. LIME：局部代理模型 · Local surrogate models

6.1 The idea

Ribeiro, Singh & Guestrin (KDD 2016) titled their paper with the question users ask: “Why should I trust you?” LIME’s intuition, in the Lab 2 slides’ words: it is much easier to approximate a black box locally, in the neighborhood of the prediction we want to explain, than globally. Formally,

where $G$ is a class of interpretable models (sparse linear models), $\mathcal L$ is a $\pi_x$-weighted squared loss, and $\Omega$ penalizes complexity (e.g., the number of nonzero coefficients). The recipe from the 2022 slides: (1) sample points around $x$; (2) get the black box’s predictions on them; (3) weight them by proximity to $x$; (4) fit the simple model on the weighted samples; (5) use it to explain.

Explanations are expressed in an interpretable representation $x' \in \{0,1\}^{d'}$: presence or absence of words for text, of super-pixels for images, of discretized feature bins for tabular data. The paper’s demonstrations are classics. A text classifier separating “Christianity” from “atheism” posts relied on header words such as “Posting” and “Host” and a name that appeared mostly in one class — tokens that would never generalize (the 2022 slides’ first example of why explanations matter). A deliberately biased husky-vs-wolf classifier relied on snow; shown the explanations, most participants in the authors’ study stopped trusting it. The paper also proposed SP-LIME, a submodular pick of a few representative explanations as a global summary.

6.2 核宽度：局部有多”局部”？

LIME 最关键、也最任意的参数是核宽度 σ。HW2 Coding Exercise 2 的设置：一维函数、以所有数据点为邻域、权重 $e^{-|x - x'|^2/d}$，在 $x = 0$ 处拟合加权线性回归。代码 03 演示 1 用 $f(x) = 0.1x^3 - x$、数据 $x \sim N(0, 2^2)$：

同一个模型、同一个点，“x 对预测的影响”从 −0.985（几乎就是导数 −1）变成 +0.209（接近全局回归）。两个答案都”对”：在 $x$ 附近，增大 $x$ 让 $f$ 减小；在整个数据范围内，增大 $x$ 让 $f$ 增大。LIME 的 width 决定了你问的是哪个问题，而它没有一个由数据或模型决定的”正确值”。lime 库表格版的默认值是 $0.75\sqrt{d}$（而且库里的核还多开了一次平方根）——一个工程约定，不是理论结论。

6.3 LIME 和 KernelSHAP：同一个回归，不同的权重

在子集空间里，LIME 和 KernelSHAP 做的是完全相同的事：在 $z' \in \{0,1\}^d$ 上拟合 $v(z') \approx \beta_0 + \sum_j \beta_j z'_j$。区别只有两点：权重（LIME 的指数核 vs Shapley kernel），以及 KernelSHAP 加了 $\sum\phi = v(N) - v(\emptyset)$ 的约束。Lundberg & Lee 正是用这一点把 LIME 统一进 SHAP 框架的。代码 03 演示 2 在第 1 讲世界的二次逻辑回归 Q 上，解释一个被拒申请人（第 5 号：p = 0.284，背景平均 0.687，要分配的总量 −0.403）：

• 宽度很小时只有”全保留”和”只去掉一个”的子集有权重，LIME → leave-one-out；宽度很大时所有子集等权，LIME → Banzhaf（均匀权重最小二乘的系数就是 Banzhaf 值，Hammer & Holzman 1992）。中间的每个宽度给出一个不同的答案，没有一个宽度满足 efficiency。
• LOO 多算了 0.110（总量的 27%）：收入低和有逾期两个不利因素在 sigmoid 里互相”放大”，单独去掉任何一个时都显得负全责。
• 首要理由在 income 和 missed 之间翻转，因为两者的贡献几乎一样大（Shapley：−0.1641 vs −0.1646）。贡献接近时，“第一理由”由方法的细节决定 —— 应该并列报告，而不是假装有一个明确的第一名。

6.4 连续 LIME：斜率 ≠ 归因

在表格数据上直接撒点（$z = x + \sigma_j \epsilon$，$\epsilon \sim N(0, I)$）时，LIME 的系数是局部斜率：“这个特征增加一个标准差，代理模型的输出变多少”。代码 03 演示 3，同一位申请人，每个宽度 20000 个扰动样本：

在这个光滑、接近线性的模型上，宽度只让所有斜率一起缩小约 25–28%，排序没有变（我预想会变，实际没有；见代码 README）。真正的教训在最后一行：LIME 说最不利的是 debt_ratio，可这位申请人的负债率几乎正好是平均水平（x̃ = +0.13），它对”为什么被拒”几乎没有贡献（φ = −0.019）。Shapley 的主要理由是逾期（2 次，x̃ = +1.30）和收入（低于平均）。income 的 LIME 斜率还是正的（收入越高越好），而它的 Shapley 值是负的（这位申请人的收入低）。

斜率回答”如果改变这个特征会怎样”（sensitivity, what-if）；Shapley 值回答”这个特征的取值让预测偏离参考水平多少”（attribution）。 两者都有用，但拿连续 LIME 的系数当拒绝理由，回答的是另一个问题。（lime 库表格版默认先把特征离散成四分位箱，系数变成”处在这个箱里”的效应，更接近归因，但又依赖箱的边界。）

6.5 稳定性：一个和流行说法相反的结果

文献常说 “LIME is unstable”（例如 Alvarez-Melis & Jaakkola 2018）。代码 03 演示 4 在 10 个被拒申请人 × 50 个种子上比较同样模型调用次数下的噪声（各特征估计值在种子之间的标准差 / 收敛答案的平均 |值|）和”最不利特征”与各自收敛答案一致的比例：

在这里，默认宽度的 LIME 比单参考点的采样 Shapley 稳定得多：宽核的回归几乎用上了所有样本，要估的目标（一个平均斜率）很容易估；而单参考点采样的每个样本只用一个参考人，不同参考人给出的贡献差别很大（§7.4）。窄核的 LIME 确实不稳定 —— “局部”恰恰是 LIME 名字里的 L。三个结论：

1. 稳定性取决于核和采样器，没有一边倒的赢家；用好的采样器（§5.2–5.3）可以大幅降低 Shapley 的噪声。
2. 更根本的是收敛到什么：采样 Shapley 收敛到一个由公理和 game 唯一确定的量；LIME 收敛到的量还取决于 width、采样分布和正则化。稳定地收敛到一个任意的目标，不比不稳定好多少。
3. 收敛之后，LIME 的”最不利特征”和 Shapley 的首要理由在 10 个人里只有 2 个相同 —— 又是”斜率 ≠ 归因”。

7. “把一个特征去掉”是什么意思 · What does it mean to remove a feature?

7.1 The value function is the explanation

Shapley’s axioms tell us how to split $v(N) - v(\emptyset)$. They say nothing about what $v(S)$ — “the model’s output when only $S$ is known” — should be. A model cannot run with missing inputs, so every method must simulate absence. The main options:

When features are independent, the interventional and conditional games coincide. All the trouble comes from correlated features.

7.2 True to the model, or true to the data?

代码 04 演示 1 用两个相关的标准正态特征（相关系数 ρ）和三个线性模型，解释 $x = (1, 1)$；线性 + 高斯时两种 game 都有精确解：

两个现象：

1. Conditional 打破了”对模型而言”的 dummy。 $f_a$ 根本不读 $x_2$，conditional 却在 ρ = 0.99 时给它 0.495 —— 几乎和 $x_1$ 一样多。原因：知道 $x_2 = 1$ 就几乎知道 $x_1 \approx 1$，“只知道 $x_2$“时模型的期望输出已经接近 $f(x)$。（形式上 dummy 公理没有被违反 —— 在 conditional 博弈里 $x_2$ 确实不是 null player；被违反的是我们真正关心的”模型不用的特征得 0”，Sundararajan & Najmi 称之为 Insensitivity。）Merrick & Taly（2020）的摘要直说：“the popular SHAP algorithm’s formulation may give substantial attributions to features that play no role in the model.”
2. Interventional 区分了 Rashomon 双胞胎。 ρ = 0.99 时 $f_b = x_1 + x_2$ 和 $f_c = 2x_1$ 在数据上几乎无法区分（差的标准差 0.14），interventional 却给出 (1, 1) 和 (2, 0)；conditional 给出几乎相同的 (1, 1) 和 (1.01, 0.99)。

Chen, Janizek, Lundberg & Lee (2020) named the two stances: interventional explanations are true to the model (what does this function do with its inputs?), conditional explanations are true to the data (which information in the data is associated with the prediction?). Janzing, Minorics & Blöbaum (2020) argued from causality that the interventional version is the right one for explaining a model: the model’s inputs are something we can intervene on — a do-operation $do(X_{\bar S} = r_{\bar S})$ on the model’s input, not on the world. For debugging, compliance and adverse-action reasons, we care about the model, so this lecture’s default is interventional. If you care about the phenomenon, the data-centric stance has merit — but then a feature the model never reads can show up in your explanation.

7.3 Off-manifold：interventional 的代价

Interventional 把 $x$ 的一部分字段和另一个人的其余字段拼在一起。字段相关时，拼出来的”申请人”可能在现实中不存在：收入 80k、存款 2k。代码 04 演示 3 用我们知道的生成公式检查这些混合输入：真实数据里只有 0.85% 被判为”不可能”，interventional 的 102,400 个混合输入中有 11.5%（中等大小的 $S$ 最严重，13%）；conditional 按相似度加权后降到 4.4%。

为什么要在乎？模型从没在这些点上训练过，它在那里的输出是任意的外推；interventional 解释有一部分是在描述模型在数据之外的行为。在我们的例子里，9 字段树模型在不可能的点上的平均输出（0.533）和可能的点上（0.564）差不多 —— 这个诚实训练的模型碰巧没有怪异行为。但没有任何东西保证这一点。Slack et al.（2020, “Fooling LIME and SHAP”）正是利用”扰动样本可以被识别出来”，构造了一个在真实数据上歧视、在扰动样本上表现无害的模型，让 LIME 和 KernelSHAP 都报告无害的解释（第 8 讲）。Permutation importance 有同样的问题（Hooker, Mentch & Zhou 2021：“unrestricted permutation forces extrapolation”）。

Conditional 留在数据上，但有自己的代价：除了 §7.2 的 dummy 问题，它很难估计。代码 04 用核加权的经验方法（Aas, Jullum & Løland 2021 的思路的简化版）：如果要求每个被条件的字段都接近 $x$，有效样本量从 |S| = 1 时的 502 塌缩到 |S| = 8 时的 9（维度灾难）；如果像本实现一样把距离按 |S| 平均，有效样本量保持在 400 左右，代价是条件很”松”（偏差）。没有免费的午餐。

7.4 The Explanation Game: the reference is part of the question

Merrick & Taly (2020) unify the interventional-style formulations with one game parameterized by a reference distribution:

KernelSHAP is $\mathcal D_{\text{ref}}$ = the input distribution; Merrick & Taly place QII at the joint-marginal distribution (by the QII paper’s own definition of set interventions it is the input distribution again, in dual form); IME is uniform. (The conditional game cannot be written this way, and it is the one that fails Insensitivity.) Their building block is the single-reference game $v_{x, r}(S) = f(z(x, r, S)) - f(r)$, which explains $f(x) - f(r)$ “relative to $r$”. Since the Shapley value is linear in the game (additivity),

Three consequences:

1. Free confidence intervals. An interventional attribution is a sample mean of single-reference attributions, so its uncertainty is the standard error of that mean. That is the interval whose coverage §5.5 checked (0.89 at m = 10, 0.95 at m = 100).
2. The mean can hide the story. Code 02 demo 4 shows the 1000 single-reference attributions behind one applicant’s mean. For income the mean is −0.056, but 64% of references give a negative value, 25% a positive one, and the interquartile range runs from −0.152 to +0.030; for missed the attribution is never negative and is exactly 0 for the 47% of references who share the applicant’s value. Merrick & Taly report a bike-sharing example in which the hour feature had a tiny mean attribution yet the largest absolute attribution in over 60% of single-reference games. They recommend looking at the distribution (variance, IQR, clustering) before trusting the mean.
3. Choose the reference on purpose. Their formulate, approximate, explain (FAE) framework makes $\mathcal D_{\text{ref}}$ a first-class argument: “the references must be consciously chosen by the explainee to obtain a specific contrastive explanation.” This follows cognitive-psychology work on how people explain surprising outcomes by contrast with normal ones. Their example: an auto-grader’s B+ should be explained against A− submissions, not against the whole pool. In their Lending Club case study, a rejected application is explained relative to accepted applications. That question gives credit score, income and debt-to-income ratio noticeably larger attributions than the default question does.

7.5 同一个申请人，五个答案

代码 04 演示 2：模型 T5 是只读 5 个正当字段的梯度提升树，但申请表有 9 个字段，所以 savings、utilization、accounts、app_minute 对它是 dummy —— 改它们绝对不会改变输出。被拒申请人第 0 号（p = 0.396）：收入 45k（均值 62.2k）、负债率 0.322（均值 0.353）、历史 4.0 年（均值 7.6）、逾期 2 次（均值 0.8）、查询 3 次（均值 1.5）。

（savings 和 app_minute 两列省略：conditional 下分别是 −0.000 和 +0.001，其他各行都是 0。）

• 零基线在表格数据上毫无意义：“收入 0、负债率 0、逾期 0 次”的人不存在，而且 0 对不同字段的含义完全不同（0 次逾期是最好的值，0 收入是最差的值）。结果：它说首要理由是负债率（−0.368），可这位申请人的负债率低于平均；它还说收入”帮了忙”（+0.215），因为 45k 比 0 高。在 20 个被拒者上（演示 4），零基线让 19 个人的首要理由都是 debt_ratio。图像的”黑图”、文本的”空串”同样不是真正中性的 —— 第 3 讲讨论 IG 的 baseline 时还会遇到。
• mean baseline vs interventional：前者和一个所有字段都取均值的”平均申请人”（并不存在，比如逾期 0.81 次）比，后者和随机的真实申请人比、再平均。对非线性模型 $f(\mathbb E X) \neq \mathbb E f(X)$，两者的 v(∅) 不同（0.736 vs 0.686），第二个理由也不同。
• conditional 给 T5 不读的字段非零功劳。对这个人它们还排不进前四；但在 20 个被拒者上，80% 的人在 conditional 的前四个理由里出现了模型不读的字段（平均 12.8% 的 |φ| 落在这 4 个字段上），所有 interventional 型定义给它们的功劳都精确为 0。
• vs approved 回答 Merrick & Taly 的 contrastive 问题：“和被批准的人相比，是什么让 TA 被拒？“v(∅) 变成 0.764，要解释的差距更大，第二、三个理由的顺序也变了。

合理的定义之间（均值基线、interventional、conditional、vs approved）首要理由的一致率是 0.85–0.90（演示 4）：多数人的主要理由不受影响，但每 10 个人里有 1–2 个会因为”和谁比”不同而得到不同的首要理由 —— 通常是前两个因素贡献接近的人。

实践建议：

8. 从局部到全局，以及信贷案例 · From local to global, and the credit case

8.1 全局解释：mean |SHAP| vs permutation importance

把局部解释汇总成全局解释，最常见的是 mean |SHAP|：对很多人的 |φ_j| 取平均（shap 库的 summary plot）；另一个有用的视图是 dependence plot：横轴是特征值、纵轴是它的 SHAP 值（HW2 Written Exercise 6）。代码 05 演示 1 把第 1 讲的 Model A / B 换成梯度提升树（同样的数据、同样的 seed）：

树模型重演了第 1 讲的故事，两种全局解释都一眼找到泄露。第 1 讲用线性模型的系数也能找到它；树模型没有系数，Shapley 仍然适用 —— 这就是 model-agnostic 的价值。

两者回答的问题不同：mean |SHAP| 是”预测平均被这个特征移动了多少”（只看模型输出，不需要标签），permutation importance 是”打乱它，准确率掉多少”（看模型和标签的关系）。A 的 income 平均移动预测 0.045，打乱它准确率却只掉 0.009 —— 有 officer_flag 在，A 的大部分决定不靠收入也能做对。一个更有原则的全局版本是对 loss 做 Shapley 分解（SAGE，Covert, Lundberg & Lee 2020）。

局部：被 A 拒绝、被 B 批准、而且实际按时还款的测试申请人有 106 人，A 给他们的首要拒绝理由（最负的 φ）全部是 officer_flag。第 1 讲的第 31 号申请人：A 的 p = 0.289，Shapley 值 officer_flag −0.449、history −0.056、income −0.039……；在 logit 空间，officer_flag 是 −2.929 —— 和第 1 讲线性 Model A 的贡献几乎一样（两个模型对这个二值泄露学到了相近的对数几率偏移）。如果银行按 Shapley 给出拒绝理由，这些人会收到”您的档案里有信贷员标记”。解释首先是 debugging 工具。

8.2 QII 解释群体差距

代码 05 演示 3：树模型 B 在部署数据上的批准率，group 0 是 0.829，group 1 是 0.594，差距 Q = 0.235。把 Q 当作 quantity of interest，做 QII：

差距几乎全部来自信用历史和收入 —— 这两个特征在两个群体里分布不同（第 1 讲 §5.2）；负债率、逾期、查询在两个群体里同分布，影响接近 0。这里 unary QII 和 Shapley-QII 差别不大，说明这几个特征对差距的作用接近可加。这是 QII 独有的能力：解释的不是一个人的预测，而是一个群体性质（disparate impact，第 5 讲）。

解释说的是模型，不是世界. 同一个演示还比较了信用历史短于 3 年的部署申请人身上，history 对 logit 的 Shapley 值和真实世界里 history 的贡献（我们有 oracle）：

真实世界里短信用历史对 group 1 几乎不代表风险（thin file，β = 0.15），对 group 0 代表很大风险（β = 0.8）。模型不知道群体，只能学一个”平均”的惩罚：对两个群体扣的分几乎一样多 —— 对 group 0 罚轻了，对 group 1 罚重了。Shapley 值忠实地报告了模型的做法，包括它对 group 1 的不公平。解释方法不会告诉你”这个依赖是错的”；判断它需要知道世界。

8.3 拒绝理由 · Adverse action reasons

美国的 Equal Credit Opportunity Act 及其实施条例 Regulation B 要求贷款机构在拒绝申请时告诉申请人”具体的主要理由”（specific principal reasons）；Reg B 的官方解释说，给出超过四个理由通常对申请人没有帮助，而且理由必须对应实际被考虑的因素。CFPB 2022 年的 Circular 2022-03 强调：使用复杂算法不能免除这个义务，“技术太复杂、太不透明”不是不合规的理由；如果依赖近似模型的解释工具，机构必须能验证这些近似的准确性。（这份 circular 已于 2025 年 5 月被撤回，但 ECOA / Reg B 本身的要求不变。）

The 2022 lecture’s conclusion was that “Shapley value based methods can be the basis for meaningful reason codes”. They can, but only after answering compared with whom?

代码 05 演示 4，树模型 B，部署数据里前 100 个被拒申请人，三种参考：

• “和谁比”首先改变的是要解释的差距有多大。“全体”和”被批准的人”之间，首要理由 97% 一致、前两个理由 95% 一致 —— 理由基本不变，只是分量更重。（我原本预期会有更大的变化；Merrick & Taly 在 Lending Club 上看到的，正是这种”分量”的变化。）
• 单一的”理想申请人”参考（业界 “points below max” 做法的类比）和另外两种的一致率低得多（首要理由 0.81–0.84，前两个理由 0.65–0.68）：它把差距推向这位申请人离”完美”最远的地方。
• 演示还打印了一封示例通知（参考 = 被批准的人）：收入 30.3k（被批准者中位数 64.6k），贡献 −0.333；负债率 0.42（0.338），−0.108；逾期 1 次（被批准者中位数也是 1），−0.035；查询 2 次（1），−0.015。后两个理由的贡献很小，逾期次数甚至和典型的被批准者一样（负贡献来自”和没有逾期的那一半人比”）。为了凑满四个理由而列出这种小贡献，会把噪声写进通知 —— 应设最小贡献阈值，并检查理由在背景重抽样下是否稳定（§5.5）。

最后，理由 ≠ 建议。“收入是主要理由”不等于”把收入提高到 X 就能获批”。后者是 counterfactual explanation / algorithmic recourse 的问题（Wachter, Mittelstadt & Russell 2017；Ustun, Spangher & Liu 2019），需要搜索最小的、申请人能做到的改变 —— 信用历史不能一夜变长。2023 年讲座的 “targeted what-ifs”（Watson et al. 2021, “Local Explanations via Necessity and Sufficiency”）就是这个方向：在扰动空间里按偏序搜索能达到目标预测的最小扰动。

9. 局限 · Limits

9.1 解释不是因果 · Explanations are about the model

An attribution explains how a model maps inputs to outputs, not how the world works. §8.2’s thin-file table is the cleanest illustration: the explanation faithfully reports that the model penalizes short credit history equally for both groups, which is false about the world. Even an interventional Shapley value is “causal” only with respect to the model’s inputs (a do-operation on what we feed the model), not with respect to the applicant’s life. If the question is about the world — “would raising this person’s income make them more likely to repay?” — you need a causal model of the world. Shapley variants that inject causal knowledge exist (asymmetric Shapley values, Frye et al. 2020; causal Shapley values, Heskes et al. 2020), but they need a causal graph that someone has to supply and defend. The 2022 lecture ended its explanation module on exactly this point, listing among its “questions to ponder”: what is the difference between causality and correlation, and is causality important for explanations?

9.2 Faithfulness and how to evaluate it

How do we know an attribution is right? The 2023 lecture lists two common evaluations and their problems. Ablate the top-attributed features and watch the prediction — but ablation creates artifacts (unrealistic inputs, §7.3). Compare with human ground truth — but the model may genuinely reason differently, and evaluators suffer from confirmation bias. The axiomatic approach is the alternative: list desirable properties, then prove a method is the unique one satisfying them. That is what §4 did for Shapley and what Lecture 3 does for Integrated Gradients. Axioms constrain how credit is divided; they cannot certify that the game asks the right question (§7). Synthetic worlds where the truth is known, like this course’s, are a third option, and that is why we use them.

The same lecture also warns that attributions are shallow: they do not explain how the network combines the features, what training data influenced the prediction (Lecture 4), or why training converged where it did. Attributions are most useful when a strict subset of the input features is what matters.

9.3 人类可理解性 · Human interpretability

Explanations are ultimately read by people, so whether one is “good” is an empirical question about people.

• Doshi-Velez & Kim (2017) distinguish three levels of evaluation: application-grounded (real users, real tasks), human-grounded (lay users, simplified tasks) and functionally-grounded (no humans; a proxy such as explanation length). Most papers only do the third.
• Lage et al. (2019), the 2022 reading “An Evaluation of the Human-Interpretability of Explanation”, gave participants explanations of varying complexity on three tasks: simulating the system’s output, verifying a suggested output, and deciding whether the suggested output’s correctness changes when the input changes (counterfactual). Introducing new concepts (“cognitive chunks”) hurt performance more than repeating existing variables did, and this pattern held across tasks and domains, which suggests some common design principles for explanation systems.
• Poursabzi-Sangdeh et al. (CHI 2021) ran pre-registered experiments with thousands of participants. A clear model with few features helped people simulate the model’s predictions, but it did not make them follow the model more when following it was beneficial. Transparency even made it harder to detect and correct the model’s sizable mistakes, apparently because of information overload.
• Kaur et al. (CHI 2020) watched data scientists use GAM and SHAP tools. Many over-trusted and misused the visualizations, and some could not accurately describe what the plots showed.

The 2023 lecture summarized the human’s role this way: humans are poor at foreseeing problems but excellent at understanding the real-world implications of specific explanations (“pen marks won’t be on X-rays in deployment”; “most words in a question should matter”). An explanation’s value lies in putting concrete evidence in front of the right person. Good visualization (for example, clipping long-tailed attributions at the 99th percentile before display) plus human thought is what turns attributions into insight.

9.4 解释可以被攻击 · Explanations can be manipulated（第 8 讲预告）

• Fooling LIME and SHAP（Slack et al. 2020）：利用扰动样本 off-manifold（§7.3），训练一个”分类器的分类器”识别扰动样本，对真实输入用歧视性的规则，对扰动样本用无害的规则 —— 解释只看到无害的那一半。
• 对梯度类解释的攻击（Ghorbani et al. 2019；Dombrowski et al. 2019）：几乎看不见的输入扰动能大幅改变归因图而不改变预测（第 3、8 讲）。
• Kumar et al.（2020, “Problems with Shapley-value-based explanations as feature importance measures”）系统地讨论了 Shapley 解释的数学问题：value function 的选择、加性的解释能力有限、不能直接回答用户关心的 what-if 问题。

这些结果的共同教训是第 1 讲 §4.1 的那句话：一个漂亮的解释可以增加信任、同时降低可信度。解释是证据，不是证书。

10. 小结 · Summary

Six takeaways:

1. An explanation is an answer to a question; say which question. Which output (logit, probability, decision, loss, group statistic), which inputs, and — above all — compared with what.
2. Marginal contributions depend on context, so average over contexts. The Shapley value is the unique average that satisfies efficiency, symmetry, dummy and additivity; “SHAP”, “Shapley-QII” and “the explanation game” share it.
3. The axioms fix how credit is divided, not what the game is. Single baseline, interventional, conditional and contrastive games give different numbers and different reasons for the same person and the same model.
4. Interventional is true to the model; conditional is true to the data. For debugging, compliance and reason codes, prefer interventional — and remember it queries off-manifold inputs.
5. Every number has error bars, from sampling coalitions and from the background sample. Report both; check that the ranking of reasons is stable.
6. Explanations describe the model, not the world, and they can be gamed. They are a debugging tool and a source of evidence for a human with domain knowledge — not a certificate of fairness or correctness.

11. 常见误区 · Common pitfalls

• “SHAP gives the Shapley value of each feature.” It gives the Shapley value of a game. Change the background, the baseline, or interventional ↔ conditional, and the numbers and the reasons change (§7.5).
• “Shapley values are unique, so they are objective.” Uniqueness holds given the game and the four axioms. The game is a modeling choice that must be justified and documented.
• “Feature X has a large Shapley value, so X causes the outcome.” It means the model relies on X for this prediction, relative to the chosen reference. The world may be different (§8.2’s thin file).
• “A zero baseline is neutral.” In tabular data zero is often the best or the worst possible value. It blamed debt_ratio for 19 of 20 denied applicants (§7.5).
• “The conditional version is more realistic, so it’s better.” It stays on the data manifold, but it can credit features the model never reads — fatal for reason codes (§7.2, §7.5).
• “LIME’s coefficients are attributions.” Continuous LIME reports local slopes. A feature can have a steep slope and zero contribution, or the opposite sign (§6.4).
• “I computed exact Shapley values, so there’s no error.” Exact over 2^d coalitions, but still an estimate over a finite background. With 100 background points the top feature matched the large-background answer only 74% of the time (§5.5).
• “One-at-a-time ablation is a fine sanity check.” It is fine for additive models and misleading whenever features interact or are redundant (§2).
• “Shapley value 0 means the feature doesn’t matter.” XOR: both features have Shapley value 0, and the function is pure interaction (§4.5). Check interaction indices.

12. 练习 · Exercises

练习 1（手算 Shapley 和 Banzhaf）. 加权投票博弈，票数 (4, 3, 2, 1)，门槛 6（总票数 10）。(a) 列出所有”获胜”的最小联盟。(b) 计算 Shapley–Shubik 指数（提示：枚举 24 个排列中每个玩家是”关键人”的次数，或用排列形式）。(c) 计算原始 Banzhaf 值并归一化。两者一样吗？有没有 dummy 玩家？

(a) 最小获胜联盟：{1, 2}（7）、{1, 3}（6）、{2, 3, 4}（6）。（{1, 4} 只有 5，{2, 3} 只有 5。） (b) 在 24 个排列里数关键人（“加入后总票数第一次 ≥ 6 的那个人”）：P1 是关键人 10 次，P2 6 次，P3 6 次，P4 2 次，所以 Shapley–Shubik = (5/12, 1/4, 1/4, 1/12) ≈ (0.417, 0.25, 0.25, 0.083)。 (c) 对每个玩家，在不含 TA 的 8 个联盟里数”TA 加入后从输变赢”（swing）的个数。P1：{2}（3 → 7）、{3}（2 → 6）、{2,3}（5 → 9）、{2,4}（4 → 8）、{3,4}（3 → 7），共 5 个（∅、{4} 加上 P1 仍然不够 6，{2,3,4} 本来就赢）；P2：{1}、{1,4}、{3,4}，3 个；P3：{1}、{1,4}、{2,4}，3 个；P4：只有 {2,3}，1 个。原始 Banzhaf = (5, 3, 3, 1)/8，归一化 = (5, 3, 3, 1)/12 ≈ (0.417, 0.25, 0.25, 0.083)。这个例子里两者恰好相同；没有 dummy（P4 在 {2, 3} 时是关键人）。可以用 shapley_tools.banzhaf_exact 和 shapley_exact 验证（输出分别是 (5, 3, 3, 1)/8 和 (5, 3, 3, 1)/12）；代码 README 的”动手改一改” 1 让你试更多门槛。

练习 2（可加模型与望远镜求和）. (a) Show that for an additive game $v(S) = c + \sum_{i \in S} a_i$, leave-one-out, add-one-in, Banzhaf and Shapley all give $\phi_i = a_i$. (b) Show that along any single ordering $\pi$, the marginal contributions $v(\text{Pre}_i(\pi) \cup i) - v(\text{Pre}_i(\pi))$ sum to $v(N) - v(\emptyset)$. (c) Use (b) to explain why permutation sampling satisfies efficiency exactly for every sample size, while Banzhaf does not satisfy it even with exact computation.

(a) 每个边际贡献 $v(S \cup i) - v(S) = a_i$ 与 $S$ 无关；任何”对情境的加权平均”（权重和为 1）都等于 $a_i$。LOO 和 add-one-in 是只放在一个情境上的权重，Banzhaf、Shapley 是两种不同的平均权重 —— 所以全部相等。这就是为什么线性模型在 logit 上”怎么解释都一样”（§2.3、§4.6）。 (b) 设 $\pi = (\pi_1, \dots, \pi_d)$，第 $t$ 步的边际贡献是 $v(\{\pi_1..\pi_t\}) - v(\{\pi_1..\pi_{t-1}\})$，相加时中间项两两抵消，剩下 $v(N) - v(\emptyset)$。 (c) 排列采样的估计是若干个排列的平均，每个排列都精确满足 efficiency，平均也满足。Banzhaf 的权重 $1/2^{d-1}$ 不对应任何”排列上的平均”：对每个玩家，各情境权重和为 1，但对所有玩家加起来不保证望远镜求和（例如 AND 的三人版本 $u_{123}$：每个玩家只在 $S =$ 其他两人时边际贡献为 1，Banzhaf = 1/4，三人之和 3/4 ≠ 1）。

练习 3（HW2 Written Exercise 4：乘积模型）. $f(x) = \prod_{i=1}^d x_i$，各特征独立、服从 $N(0, 1)$。用 interventional value function（背景 = 特征分布），求任意点 $x$ 处的 SHAP 值的闭式解。这个答案符合直觉吗？以 $d = 2$、$x = (100, 0.01)$ 为例讨论。

$v(S) = \mathbb E[\prod_{i \in S} x_i \prod_{j \notin S} X_j] = \prod_{i \in S} x_i \prod_{j \notin S}\mathbb E[X_j] = 0$ unless $S = N$, where $v(N) = \prod_i x_i$. So $v = (\prod_i x_i)\cdot u_N$ is a scaled unanimity game, and $\phi_i = \frac{1}{d}\prod_j x_j$: every feature gets the same share, regardless of its own value. At $x = (100, 0.01)$, $f = 1$ and both features get 0.5, although the first is 100 standard deviations from its mean and the second sits almost at its mean. The discomfort comes from the model being pure interaction (the only nonzero dividend is on $N$), and the Shapley value splits interactions equally. Lessons: (i) report interactions separately (§4.5); (ii) “fair division” is not the same as a human’s intuition of “importance”. Explaining $\log|f| = \sum_i \log|x_i|$ instead makes the model additive, and the attributions then reflect each feature’s size — once more, which output you explain is a choice.

练习 4（相关特征的闭式解）. Let $(X_1, X_2)$ be standard bivariate normal with correlation $\rho$, $f(x) = x_1$, and explain $x = (a, b)$. (a) Compute the interventional Shapley values. (b) Compute the conditional Shapley values (use $\mathbb E[X_1 \mid X_2 = b] = \rho b$). (c) Check against §7.2 for $a = b = 1$, $\rho = 0.9$. (d) For which $(a, b)$ does the conditional game give feature 2 a negative attribution, and what does that mean for an applicant?

(a) $v(\emptyset) = 0$，$v(\{1\}) = a$，$v(\{2\}) = 0$，$v(N) = a$ ⇒ $\phi = (a, 0)$。 (b) $v(\emptyset) = 0$，$v(\{1\}) = a$，$v(\{2\}) = \rho b$，$v(N) = a$ ⇒ $\phi_1 = \tfrac12 a + \tfrac12(a - \rho b) = a - \rho b/2$，$\phi_2 = \tfrac12\rho b + \tfrac12(a - a) = \rho b/2$。 (c) $a = b = 1$、$\rho = 0.9$：(0.55, 0.45)，与代码 04 演示 1 一致。 (d) $\phi_2 = \rho b/2 < 0$ 当 $\rho b < 0$，例如正相关且 $b < 0$。含义：模型根本不读 $x_2$，但 conditional 解释会告诉一个 $x_2$ 偏低的申请人”$x_2$ 让你的分数变低了”—— 一个模型从未考虑过的”理由”（§7.5 的 savings、utilization 正是这种情况）。

练习 5（KernelSHAP 在 d = 2 时）. For $d = 2$, the only coalitions besides $\emptyset$ and $N$ are $\{1\}$ and $\{2\}$, with equal Shapley-kernel weights. Solve $\min\,[(v_1 - v_0 - \phi_1)^2 + (v_2 - v_0 - \phi_2)^2]$ subject to $\phi_1 + \phi_2 = v_{12} - v_0$ and show the solution is the Shapley value. Then explain what goes wrong if you drop the constraint and instead fit an intercept, as LIME does.

拉格朗日条件：$\phi_1 - (v_1 - v_0) = \phi_2 - (v_2 - v_0)$，即 $\phi_1 - \phi_2 = v_1 - v_2$。与约束联立：$\phi_1 = \tfrac12(v_{12} - v_0 + v_1 - v_2) = \tfrac12(v_1 - v_0) + \tfrac12(v_{12} - v_2)$，正是 $d = 2$ 的 Shapley 公式（两个情境各占一半）；$\phi_2$ 同理。 去掉约束、加截距后，LIME 拟合的是 4 个点 $(z', v)$：(0,0)→$v_0$、(1,0)→$v_1$、(0,1)→$v_2$、(1,1)→$v_{12}$，权重由核决定（离 (1,1) 越近越大）。三个参数、四个点，只有在 $v$ 可加（交互 $v_{12} - v_1 - v_2 + v_0 = 0$）时才能完美拟合；否则残差的分配取决于权重：权重集中在 (1,1) 及其邻居时得到 LOO（$\beta_i = v_{12} - v_{N \setminus i}$），等权时得到 Banzhaf —— $d = 2$ 时 Banzhaf 恰好等于 Shapley，但从 $d = 3$ 起就不同了（§6.3 的表）。截距吸收了一部分总量，所以系数之和一般 ≠ $v_{12} - v_0$。

练习 6（为一家银行写解释政策）. 你负责一家银行的信贷模型（梯度提升树，30 个字段）的拒绝理由生成。写一页政策，回答：(a) 解释哪个输出？(b) 用什么 value function 和参考分布？为什么？(c) 背景样本多大、怎样报告不确定性？(d) 什么情况下一个特征不列为理由（最小贡献、近似并列、不可改变的特征）？(e) 怎样检查模型没有用”不该用”的字段？(f) 哪些问题这套解释回答不了？

(a) Explain the approval probability: applicants understand “this lowered your chance of approval”; the logit is fine for internal audits; the 0/1 decision is too coarse (§1.2, §8.1). (b) Interventional, with the reference = applicants approved in the last 12 months. This is the contrastive question “how do you differ from people who were approved?” (§7.4–7.5). Never a zero baseline. Not conditional, because it can credit fields the model does not read, which conflicts with “reasons must be factors actually considered” (§7.2). (c) With 30 fields, exact enumeration is impossible: use interventional TreeSHAP or paired KernelSHAP; a background of several hundred applicants; intervals from the standard error of single-reference attributions; each quarter, re-sample the background and check how often the top reason stays the same (§5.5). (d) List only negative contributions above a threshold (say 10% of the total negative contribution), at most four; when two reasons’ intervals overlap, report them as tied; handle immutable or protected attributes (age, etc.) according to the law. (e) Run global mean |SHAP| and permutation importance on all historical decisions to look for leaked fields (Lecture 1’s officer_flag, §8.1) and proxies for protected attributes; decompose group approval-rate gaps with QII (§8.2) and hand the result to the fairness analysis of Lecture 5. (f) The policy does not answer “what would get me approved?” (that needs recourse methods), nor “does this feature cause default in the real world?” (explanations are about the model, §9.1). It does not prove the model is fair, and explanations can be manipulated (§9.4), so the model itself must be audited too.

代码相关的练习（“动手改一改”）见 code/lecture02/README.md。

13. 延伸阅读 · Further reading

本讲对应的原课材料

• CS329T Spring 2022, Week 2 “Explanations” slides（Datta & Mitchell, Apr 5 / Apr 12, 2022）、Homework 2 Part I、Lab 2；Fall 2023 “Interpretability” lecture（Datta, Mitchell & Taly）。

核心论文（2022 年的阅读材料）

• Ribeiro, Singh & Guestrin, “‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier”, KDD 2016（LIME）。
• Datta, Sen & Zick, “Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems”, IEEE S&P 2016（QII）。
• Lundberg & Lee, “A Unified Approach to Interpreting Model Predictions”, NeurIPS 2017（SHAP、KernelSHAP）。
• Lage, Chen, He, Narayanan, Kim, Gershman & Doshi-Velez, “An Evaluation of the Human-Interpretability of Explanation”, 2019。

合作博弈论

• Shapley, “A Value for n-Person Games”, 1953；Shapley & Shubik, “A Method for Evaluating the Distribution of Power in a Committee System”, 1954；Young, “Monotonic Solutions of Cooperative Games”, 1985；Grabisch & Roubens, “An Axiomatic Approach to the Concept of Interaction among Players in Cooperative Games”, 1999。

计算

• Štrumbelj & Kononenko, “An Efficient Explanation of Individual Classifications using Game Theory”, JMLR 2010；Castro, Gómez & Tejada, “Polynomial Calculation of the Shapley Value Based on Sampling”, 2009；Mitchell, Cooper, Frank & Holmes, “Sampling Permutations for Shapley Value Estimation”, JMLR 2022。
• Covert & Lee, “Improving KernelSHAP: Practical Shapley Value Estimation via Linear Regression”, AISTATS 2021。
• Lundberg, Erion & Lee, “Consistent Individualized Feature Attribution for Tree Ensembles”, 2018；Lundberg et al., “From Local Explanations to Global Understanding with Explainable AI for Trees”, Nature Machine Intelligence 2020（TreeSHAP、SHAP interaction values）。
• Van den Broeck, Lykov, Schleich & Suciu, “On the Tractability of SHAP Explanations”, AAAI 2021。

Value function 与参考分布

• Merrick & Taly, “The Explanation Game: Explaining Machine Learning Models Using Shapley Values”, CD-MAKE 2020（arXiv 1909.08128）。
• Sundararajan & Najmi, “The Many Shapley Values for Model Explanation”, ICML 2020。
• Chen, Janizek, Lundberg & Lee, “True to the Model or True to the Data?”, 2020；Janzing, Minorics & Blöbaum, “Feature Relevance Quantification in Explainable AI: A Causal Problem”, AISTATS 2020。
• Aas, Jullum & Løland, “Explaining Individual Predictions when Features are Dependent”, Artificial Intelligence 2021；Frye, Rowat & Feige, “Asymmetric Shapley Values”, NeurIPS 2020；Heskes et al., “Causal Shapley Values”, NeurIPS 2020。
• Covert, Lundberg & Lee, “Explaining by Removing: A Unified Framework for Model Explanation”, JMLR 2021；同一组作者的 SAGE（“Understanding Global Feature Contributions with Additive Importance Measures”, NeurIPS 2020）。

局限、评估与攻击

• Doshi-Velez & Kim, “Towards a Rigorous Science of Interpretable Machine Learning”, 2017；Poursabzi-Sangdeh et al., “Manipulating and Measuring Model Interpretability”, CHI 2021；Kaur et al., “Interpreting Interpretability”, CHI 2020。
• Kumar, Venkatasubramanian, Scheidegger & Friedler, “Problems with Shapley-value-based Explanations as Feature Importance Measures”, ICML 2020；Slack et al., “Fooling LIME and SHAP”, AIES 2020；Hooker, Mentch & Zhou, “Unrestricted Permutation Forces Extrapolation”, Statistics and Computing 2021。
• Wachter, Mittelstadt & Russell, “Counterfactual Explanations without Opening the Black Box”, 2017；Ustun, Spangher & Liu, “Actionable Recourse in Linear Classification”, FAT* 2019。
• 法规：12 CFR §1002.9（Regulation B 的 adverse action 通知）及其官方解释；CFPB Circular 2022-03（2025 年撤回）。

14. 下一讲预告 · Next lecture

Lecture 3：深度网络的归因（Attribution for Deep Networks）。

本讲的方法都是 model-agnostic 的：只调用 $f$，代价是 $2^d$ 个子集或大量采样。一张 224×224 的图像有 15 万个输入，一段文本有几百个 token —— 枚举子集毫无希望，而 LIME 的”超像素”又把问题交给了分割算法。神经网络给了我们一个免费的东西：梯度。第 3 讲从 saliency map 和 gradient × input 出发，看它们为什么会”饱和”；然后讲 Integrated Gradients：沿着从基线到输入的直线积分梯度。它满足 completeness，并且是满足一组公理（sensitivity、implementation invariance、linearity、completeness、symmetry）的唯一路径方法 —— 历史上它就是合作博弈论里 Shapley 值的连续版本 Aumann–Shapley 值。本讲 §7 的问题会原样回来：IG 的基线（黑图？模糊图？噪声？）该怎么选。我们还会讲内部神经元的影响（influence-directed explanations）、概念级解释，以及怎样检验一个归因方法是否靠谱：deletion 曲线和 sanity checks。