Lecture 16 · 以评估为驱动的优化与课程总结 · Evaluation-Driven Optimization & Course Synthesis

对应 Maps to：Fall 2024 Nov 12 两位客座讲者 —— Omar Khattab（DSPy；论文 Khattab et al. 2023, “DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines”）与 Mert Yuksekgonul（TextGrad；论文 Yuksekgonul et al. 2024, “TextGrad: Automatic ‘Differentiation’ via Text”）—— 这两场没有公开 slides，本讲 §2–§3 依据他们的论文写成。Autumn 2025 Oct 23 Trung Vu（Bespoke Labs）“Reinforcement Learning for Agents”（slides：环境的构成与好环境的标准、环境的构造方法、policy gradient / GRPO、agent RL 的难点；阅读：Bespoke “Improving Multi-Turn Tool Use with Reinforcement Learning”、Goldie et al. 2025 “Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use”、SWE-bench、SWE-smith、ARE/Gaia2、OSWorld、Terminal-Bench、DAPO、ScaleRL；选读 OpenThoughts、NaturalThoughts）；Oct 28 Lakshya Agrawal “GEPA: Reflective Evolution of Compound AI Systems”（slides + 论文 Agrawal et al. 2025, “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning”）。多模态：Fall 2024 Oct 29 Huaxiu Yao “Data-Centric Preference Optimization for Multimodal Foundation Model Alignment”（slides；LURE、POVID、CSR 三篇论文）；Fall 2023 Lecture 16 第 2 部分 Anupam Datta “Evaluating MultiModal RAGs”（slides）；Fall 2024 Nov 14 与 Autumn 2025 Nov 13 Deepak Ramachandran 关于多模态前沿模型的讲座没有公开 slides，本讲不总结其内容。§9 的课程总结综合了第 1–15 讲和 Autumn 2025 第 1、4、7 讲的 slides。 配套代码 Code：code/lecture16/（只用 Python 标准库；5 个脚本 + 2 个工具模块；01 约 16 秒，02 约 5 秒，其余都在 2 秒以内） 预计阅读时间：3–3.5 小时（含运行代码）

0. 本讲导航 · How to read this lecture

0.1 Learning goals

After this lecture you should be able to:

1. Treat an LLM program as an optimizable object: name its parameters (instructions, demonstrations, module structure, weights), what signal each kind of optimizer consumes, and what a “rollout” costs.
2. 解释 DSPy 的三个想法（signature、module、根据 metric “编译” prompt 的 optimizer）、BootstrapFewShot 为什么只需要最终输出的标签，以及它会系统性地缺少哪些示例。
3. Explain TextGrad’s analogy with backpropagation — what plays the role of the gradient, the chain rule, the optimizer step — and where the analogy breaks.
4. 讲清楚 GEPA 的三个成分（reflection、genetic、Pareto）各自解决什么问题，以及”每次 rollout 带回多少信息”为什么决定了样本效率；能说出什么情况下它不占优。
5. Describe an RL environment for agents (user, LLM, tools, state, grader), the criteria for a good one, how GRPO turns rewards into updates, and why groups with identical rewards carry no signal.
6. 用 LURE / POVID / CSR 的例子说明多模态幻觉的来源，以及”偏好数据从哪里来”如何决定了优化是在纠错还是在钻空子。
7. Recognize Goodhart’s law in four forms — dev-set overfitting, judge exploitation, reward hacking, metric-side shortcuts — and apply the mitigations: untouched test sets, noise floors, re-validating evaluators on optimized outputs, audit sets, cheating-proof graders, multiple runs.
8. 把整门课串起来：六个透镜在 model → application → agent 三个阶段的形式、七条反复出现的方法论教训、一个可以直接用于课程项目的 trust requirements 清单。

0.2 先修与本讲范围 Prerequisites & scope

本讲有两部分。Part A（§1–§8） 讲”有了评估之后怎样改进”：prompt 优化（DSPy、TextGrad、GEPA）、RL、多模态偏好优化，以及优化如何反过来”刷”评估。Part B（§9） 是整门课的总结。

0.3 本讲地图 Map

Part A  以评估为驱动的优化
§1  从"会评估"到"会改进"：LLM 程序的参数；模拟世界 opt_world           ← 代码 01 演示 1
§2  DSPy：signature、module、bootstrap 示例、编译                      ← 代码 01 演示 3
§3  TextGrad：用文字做反向传播                                         ← 代码 01 演示 3（贪心反思）
§4  GEPA：反思、谱系、Pareto；样本效率                                 ← 代码 01 演示 2–4
§5  RL for agents：环境、GRPO、数据筛选、reward hacking                ← 代码 03
§6  多模态：物体幻觉、POVID、CSR、多模态 RAG 的评估                     ← 代码 04
§7  Goodhart again：四种"刷分"与对策                                   ← 代码 02
§8  最终系统的可信成绩单                                               ← 代码 05
Part B  课程总结
§9  六个透镜 × 三个阶段；七条方法论；项目清单；开放问题
§10–§14  小结、误区、练习、延伸阅读、课程之后

0.4 Notation

本讲不报告群体差距，所以没有第 5、6 讲那样的符号约定问题；“按题型拆开的 EM”扮演子群体的角色。

0.5 关于数字 · About the numbers

The code for this lecture calls no real LLM. opt_world.py is a simulated world: a fictional knowledge base about the Lumen Archipelago, five question types, and a “simulated LLM” whose chance of answering correctly is determined by which instruction clauses and which demonstrations are in the prompt. The effect of each clause on each question type is a known parameter written in a table. Because the truth is known, we can compute exactly the expected EM of any prompt on the test set (the “true EM”), and therefore check how good the prompt chosen by each optimizer really is, and how much each evaluator gets gamed.

所以：数字说明机制，不代表真实 LLM 的表现。In a real system nobody knows the effect of an instruction, interactions are messier, and feedback is noisier. Results that differed from the plan are kept as they came out and logged under “关于实验结果” in the code README (there are several). All scripts use only the standard library and fixed seeds, and print byte-identical output on Python 3.9 and 3.13; every comparison between optimizers reports the mean ± standard deviation over multiple runs.

Part A · 以评估为驱动的优化 · Optimization driven by evaluation

1. 从”会评估”到”会改进” · From evaluation to improvement

1.1 The course’s last step

The Autumn 2025 course description compresses the whole course into one idea (paraphrased): a prototype is easy to build; making it useful and reliable takes iterative improvement guided by clear evaluation metrics. 第 1 讲 §8 画过这个循环：

define（形式化性质）→ measure（带不确定性地测）→ improve（改模型 / 数据 / 系统）→ re-measure（在新数据上再测）

Lectures 2–15 were almost entirely about the first two steps: a definition and a measurement for each lens. This lecture is about the third step — improve — and specifically about automated improvement: an optimizer takes your metric and searches for a better program by itself.

Autumn 2025 Lecture 7 (“Deep Dive into LLMs”) gave the reason to understand how models are trained: the failure modes they bring into agents inform evaluation, and understanding why they succeed and fail informs optimization. Lecture 4’s advice for projects is the same loop: identify the agents a workflow needs → define a planner and an orchestrator → use LLM judges and other techniques as evaluation metrics → iterate until the agent is reasonable and reliable on a meaningful task.

这对”可信”有两层意义：

1. The good news: if a trust property can be measured, it can be optimized. In §8, writing “resist prompt injection” into the objective makes the optimizer discover a defensive instruction on its own.
2. The bad news: the optimizer will exploit every flaw in the metric. In 第 1 讲 §8.3 the judge was gamed once, by hand, by making answers longer; an optimizer running thousands of rollouts will systematically find every bias the judge has (§7). Once an evaluation becomes an optimization target, the trustworthiness of the evaluation bounds the trustworthiness of the system.

1.2 LLM 程序是可优化的对象 · What can be optimized

The GEPA talk calls today’s LLM applications compound AI systems: not a single model but “model + retrieval + tool calls + guardrails + control flow”. Such a system has many knobs:

The GEPA slides start from sample efficiency. The standard way to teach a model a new task is to update its weights — trillions of tokens for pretraining, tens of thousands of labeled examples for SFT, hundreds of thousands of rollouts for RL on math or coding. When rollouts are expensive (each one runs a slow workflow, or the metric itself is costly) and domain data are scarce, editing the prompt in text space may be the cheaper way to learn.

1.3 本讲的模拟世界 · The simulated world

To compare against the truth, scripts 01–03 and 05 share one simulated world (opt_world.py):

• Knowledge base: 48 fictional people (birth towns; 16 have nicknames) and 40 organizations (founding years; 10 have no recorded founder). Retrieval is simplified to “fetch the documents of the entities named in the question”.
• Five question types (share of the test set): lookup, one hop (0.30); multihop, two hops (0.25, “the founder’s birth town”); date (0.20, the answer must be a 4-digit year); unanswerable (0.15 — worded exactly like multihop, but the KB has no founder, so the correct output is UNKNOWN); alias (0.10, the nickname must be resolved first).
• Simulated LLM: for question $e$ (type $t$, difficulty $d_e \sim \mathcal N(0, 0.8^2)$) the content logit is $z = \text{BASE}[t] + \sum_{c\in I}\text{EFFECT}[c][t] + \text{demo bonus} - d_e$, and the content is right with probability $\sigma(z)$. On top of that come two format errors (with cot or elaborate but no “Answer:” line the answer is buried in the reasoning; dates written as “in the spring of 1871”) and false abstention (when the prompt allows UNKNOWN, some answerable questions also get UNKNOWN).
• Instruction space: 12 clauses that matter (decompose, cot, final_tag, date_format, abstain, alias, concise, confident, elaborate, persona, cite, spotlight), plus abstain_strict, reachable only by rewriting abstain after reading feedback. The “large space” adds 36 platitudes with no effect at all (“Take a deep breath.”, “Double-check your work.”, …), standing in for the real world, where the sentences you could put in a prompt are endless and nearly all irrelevant.
• Three evaluators: gold EM plus one sentence of textual feedback (“stopped after the first hop”, “expected a 4-digit year”, …); an LLM judge that does not see the reference (prefers long answers +0.9 and a confident tone +0.6, treats UNKNOWN as unhelpful −1.2); groundedness and prompt-injection checks.

代码 01 演示 1 先看这个世界本身：

天花板是 7 条子句：decompose + cot + final_tag + date_format + abstain_strict + alias + cite。随机写 1–6 条指令呢？12 条的空间里中位数 0.530、90% 分位 0.679；48 条的空间里中位数 0.489（和什么都不写一样）、90% 分位 0.570。

最重要的是，一次 rollout 带回来的不只是一个 0：

[multihop] In which town was the founder of Corvel Group born?
   回答：Sefa Ardent → EM 0
   反馈：The question asks for the founder's birth town, but the response gave the founder's
         name (Sefa Ardent); it stopped after the first hop.
[date] In what year was Corvel Group founded?
   回答：in the spring of 1789 → EM 0
   反馈：The expected answer is the 4-digit year 1789; the response wrote 'in the spring of
         1789', which does not match exactly.

一个标量只告诉你”错了”；一句诊断告诉你错在哪、往哪改。这是本讲前半部分的核心变量。

1.4 三类优化器，一个关键量 · Three families, one key quantity

The key quantity is how much information each rollout returns relative to how large the search space is. A scalar 0/1 carries at most one bit; a diagnosis like “stopped after the first hop” can point directly at the one instruction (out of many) that fixes the failure. Keep this in mind — it explains both when reflective methods win (§4.6) and when they don’t.

2. DSPy：把 prompt 编译出来 · Programming, not prompting

2.1 Motivation

Khattab et al.（2023，ICLR 2024）的出发点是：LLM 流水线通常由手写的长 prompt 模板拼起来，这些模板是”试出来的”，换一个模型、换一个任务、改一个模块就得重新试。DSPy 把这个过程类比为早期的神经网络：人们曾经手工调特征，后来改成声明结构、让优化器学参数。DSPy 想对 prompt 做同样的事：

• 你声明每个 LLM 调用要做什么（输入、输出），而不是写出它的 prompt；
• 一个编译器（optimizer，早期叫 teleprompter）根据你给的 metric，自动生成指令和示例。

2.2 Signatures, modules, programs

• Signature: a declarative input/output spec such as "question, context -> answer". It says what to do, not how to prompt.
• Module: turns a signature into a calling strategy — Predict (answer directly), ChainOfThought (reason, then answer), ReAct (a loop of reasoning and tool calls). Modules have parameters: their instructions and demonstrations.
• Program: ordinary Python control flow chaining modules, e.g. multi-hop QA: generate a search query → retrieve → generate another query → retrieve → answer.

Roughly like this (a sketch, not the exact API):

class MultiHopQA(dspy.Module):
    def __init__(self):
        self.gen_query = dspy.ChainOfThought("question, context -> search_query")
        self.answer = dspy.ChainOfThought("question, context -> answer")
    def forward(self, question):
        context = []
        for hop in range(2):
            q = self.gen_query(question=question, context=context).search_query
            context += retrieve(q)
        return self.answer(question=question, context=context)

2.3 Optimizers：怎样”编译”

The key trick is BootstrapFewShot: run the program (the “teacher”, by default the zero-shot program itself) on training inputs, and keep every trace whose final output passes the metric as a demonstration — including the inputs and outputs of the intermediate modules. This is why DSPy needs labels only for the final output: you never have to hand-write what the query generator should have produced at hop 1; a successful trace shows it.

On top of that:

• BootstrapFewShotWithRandomSearch：生成多组示例组合，在开发集上逐个评估，留最好的。
• MIPROv2（Opsahl-Ong et al. 2024）：用一个 LLM 根据数据摘要、程序结构和示例提出候选指令，再用贝叶斯优化在”指令 × 示例”的组合空间里搜索。
• BootstrapFinetune：把 bootstrap 出来的轨迹拿去微调一个小模型（从 prompt 到权重）。

论文报告：几分钟的编译，就让 GPT-3.5 和 llama2-13b-chat 自举出的流水线比标准的 few-shot prompting 一般高出 25% 和 65% 以上；编译后的小模型（770M 参数的 T5）也能和为 GPT-3.5 手写的专家 prompt 链相竞争。

2.4 In our world

代码 01 的 “DSPy-style bootstrap + RS” 实现了上面的骨架：先用空 prompt 当老师，在训练题上跑，收集最多 12 个通过 EM 的轨迹当示例；再在”随机的 1–6 条指令 × 随机的 1–4 个示例”上随机搜索（比 MIPROv2 的提议和贝叶斯优化朴素得多）。结果（20 次运行的平均真实 EM）：

这和我们计划的故事不一样：在小空间里，DSPy 式的方法在几乎每个预算上都是最好的（§4.6 的完整表格）。原因有两个：示例本身就是很强的修补（一个 date 示例展示了”答案只写年份”，部分起到了 date_format 指令的作用），而 12 条指令的组合只有 4096 种，随机提案很容易碰到好组合。到了大空间，随机提案大多是没用的套话，它就掉到了 0.741。

另一个值得注意的现象是示例的偏斜。20 次运行合计收集到的 240 个示例里：lookup 0.48、multihop 0.23、date 0.16、unanswerable 0.07、alias 0.05（测试集比例是 0.30 / 0.25 / 0.20 / 0.15 / 0.10）。老师只能示范它已经会做的题：最需要示范的题型（空 prompt 在 unanswerable 上只有 0.27）恰好最少。真实系统里，这意味着 bootstrapped 示例会放大训练分布里”容易的部分”，这是一个值得在项目里检查的偏差。

2.5 The trust angle

DSPy makes one thing explicit that hand-prompting hides: the metric is the specification. Whatever the metric rewards, the compiled program will do — including its loopholes. A compiled prompt is also tuned to a particular dev set and model; when either changes, re-compile and re-evaluate on fresh data. 第 1 讲 §1.3 的教训（i.i.d. 测试集只对”同一个数据流程”负责）在这里变成了：编译出来的 prompt 只对它的开发集和那个模型负责。

3. TextGrad：用文字做反向传播 · Automatic differentiation via text

3.1 The analogy

Yuksekgonul et al. (2024; the journal version appeared in Nature in 2025 as “Optimizing generative AI by backpropagating language model feedback”) carry PyTorch’s autograd over to text:

The interface mimics PyTorch (Variable, loss.backward(), optimizer.step()). The applications in the paper are deliberately diverse: GPT-4o’s zero-shot accuracy on Google-Proof QA from 51% to 55%; a 20% relative gain on LeetCode-Hard solutions; better prompts for reasoning tasks; druglike small molecules with favorable in silico binding; radiotherapy treatment plans.

3.2 Instance optimization vs prompt optimization

TextGrad draws an important distinction:

• Instance optimization: the variable is the solution to one problem (a piece of code, an answer), with feedback from test cases or an evaluator. This is test-time search; nothing has to generalize.
• Prompt optimization: the variable is the system’s instruction, which must improve on new problems — so feedback has to be collected over batches of training problems and the result validated on a dev set.

The second is the problem DSPy and GEPA address; the first is closer to “many attempts with feedback” (the GEPA slides also list inference-time search, e.g. NPU / CUDA kernel generation, as a use case).

3.3 Where the analogy breaks

A textual “gradient” has no magnitude and no guarantee of descent; it is whatever the critic LLM writes. Two consequences:

• You still need validation. A proposed update must be checked on held-out examples before it is kept — otherwise you are doing a random walk guided by a persuasive critic.
• Greedy refinement can stall. Always refining the current best prompt is like gradient descent without restarts: it gets stuck when every local edit fails validation.

代码 01 的 “greedy reflection (TextGrad-style)” 就是这种做法：总是挑当前最好的程序，读它在 3 道训练题上的反馈，提出一个修改，小批量上变好才接受。它在小预算时很快 —— 大空间 600 次 rollout 时 0.698，是所有方法里最高的 —— 但之后几乎停住：小空间里从 1200 到 4800 次都是 0.755–0.756，各次运行之间的标准差 ±0.079。当前最好程序的每一个一步修改都过不了小批量检验时，它就卡住了。下一节的 Pareto 选择就是针对这个问题的。

4. GEPA：反思式的 prompt 进化 · Reflective prompt evolution

4.1 The problem GEPA attacks

GEPA slides 开头的一张图：RL with verifiable rewards 运行一条 rollout（推理、调用工具、读结果、再推理……），最后只拿到一个 0 或 1，再把这一个数通过梯度下降传回去。

slides 上这一页的大意是：每条 rollout 里其实有很多信息，我们却只取了一个 0/1 分数 —— 能不能把其余的信息也用上？

于是 GEPA 把”RL in text space”定义为：每次 rollout 拿回的不是 reward，而是 ⟨score, text feedback⟩。

4.2 三个想法 · Three insights

4.3 The algorithm

Following the slides and the paper:

Input: training data, a system Φ with ≥ 1 prompts, a metric μ (with feedback μ_f)
Split data into D_feedback (minibatches for learning) and D_pareto (validation for selection)
Pool ← {Φ₀}; evaluate Φ₀ on every item of D_pareto
Repeat until the rollout budget is spent:
    1. Select a candidate: build the per-instance Pareto front over D_pareto
       (for each item, the candidates with the best score; drop strictly dominated ones),
       sample a candidate with probability ∝ #items on which it is a best.
    2. Pick a module (round-robin) and a minibatch of 3 items from D_feedback;
       run the candidate, collecting scores, traces and feedback text.
    3. Reflect: an LLM reads the traces + feedback and proposes a new instruction for that module.
       (Or, occasionally, merge: combine the best module versions from two lineages.)
    4. Run the child on the same minibatch; if its score is strictly better,
       evaluate it on all of D_pareto and add it to the pool.
Return the candidate with the best average score on D_pareto.

Two details matter for trust. First, most of the rollouts go to validation, not learning — the paper notes this, and in our run 79% of rollouts were spent evaluating accepted candidates on $D_\text{pareto}$. Second, the final selection is a max over many candidates on a fixed dev set — the winner’s curse of 第 1 讲 §8.2 applies in full (§7.3).

为什么 Pareto 而不是”总是挑最好的”？论文的消融（Qwen3-8B 的四个任务合计）：总是挑当前最好的候选（SelectBestCandidate）得 54.89，Pareto 选择得 61.28；作者的解释是前者会”卡在一个候选上、耗尽预算”——正是 §3.3 的现象。

4.4 Results from the paper and slides

• vs RL（GRPO）：基线是 Qwen3-8B + LoRA 的 GRPO，固定 24,000 次 rollout。论文第一版（2025 年 7 月）报告 GEPA 在 HotpotQA、IFBench、HoVer、PUPA 上平均高约 10%、最多 20%，rollout 最多少 35 倍（例如 IFBench 上 678 次就达到了测试集上的最优）；2026 年 2 月修订版在六个任务上报告的平均优势是 6%。slides 上对应的一页写着 “24000 rollouts!” vs “600 rollouts!”。
• vs MIPROv2：两种模型上都高出 10% 以上。四个任务的合计分（Qwen3-8B）：基线 48.85，MIPROv2 55.11，GRPO 51.14，GEPA 61.28。
• 用例（slides）：泛化到新数据的 prompt 学习（HotpotQA、AIME）；inference-time search（AMD NPUEval 的 NPU kernel 生成，slides 报告约 7 倍的性能、>70% 的向量利用率）；agent 架构搜索（把 agent 代码当作被进化的文本，ARC-AGI 从 44% 到 49.5%）；对抗 prompt 搜索（把奖励取反，§7.1）。

4.5 In our world: one run, step by step

代码 01 演示 2 展开一次 GEPA 运行（小空间、2400 次 rollout、开发集 60 题）。候选树的前几步：

Three things are worth seeing:

1. Most edits come straight from the feedback. Along #0 → #3 → #4 → #5 → #6, four “read the failure, apply the fix” steps take true EM from 0.489 to 0.775.
2. A stepping stone. #10 adds cot and its dev score drops from 0.650 to 0.533 (the reasoning buries the answer, so EM cannot extract it); but its feedback immediately becomes “no separate final answer line”, the next step adds final_tag, and the result is the best candidate of the run, #11 (dev 0.850, true EM 0.823). A method that keeps only the current best would have thrown #10 away.
3. The Pareto front is bigger than you might think. Here it contains all 31 candidates: scores are 0/1 and responses are random, so almost no candidate’s set of solved items is a strict subset of another’s. The selection weights are then ≈ “how many dev items it solved” (24–51), and Pareto selection behaves like score-weighted exploration. The dev-set champions of the five question types are #1, #11, #17, #9 and #20 — five different programs.

4.6 In our world: sample efficiency

代码 01 演示 3：7 种优化器 × 2 种指令空间，各跑 20 次（不同的训练 / 开发集抽样和种子），每格是给定 rollout 预算时挑出的 prompt 的平均真实 EM（± 标准差见代码输出）：

How to read it:

• In the small space, reflection does not dominate. All methods end between 0.74 and 0.82; DSPy-style bootstrap + random search is best or near-best at every budget, and even the scalar policy gradient reaches 0.808. With only 2¹² instruction sets and demonstrations as cheap, powerful patches, one bit per rollout is not a big handicap.
• In the large space, the blind methods collapse and the reflective ones barely move. Random search falls to 0.634, GEPA without text to 0.611, scalar policy gradient to 0.747 (and it hasn’t turned on a single clause after 600 rollouts); GEPA and GEPA + merge stay at 0.791 / 0.806. The value of feedback grows with the size of the search space — which is the realistic regime: the set of sentences you could put in a prompt is effectively infinite.
• Greedy vs Pareto：贪心反思起步最快（大空间 600 次时最高），但会停住；Pareto 在大预算时更好、方差也更小（4800 次：小空间 ±0.036 vs ±0.079，大空间 ±0.042 vs ±0.062）。
• Merge 稳定地小幅为正（4800 次：+0.014 和 +0.015），和论文报告的约 2% 的平均提升同一个量级。
• 每种方法”开发集上报告的分数”都比真实 EM 高 0.06–0.10（例如大空间 GEPA：0.885 vs 0.791）。这就是赢家诅咒：报告的数字必须来自没参与选择的测试集。

演示 4 把”文字反馈值多少”直接变成一个旋钮：反思读懂反馈、提出对症修改的概率 $q$（其余时候随机修改）。大空间、2400 次 rollout、20 次运行：

曲线前段平、后段陡：反思一半时间在瞎改，小批量检验 + 开发集选择也能把坏修改挡在外面；几乎全是盲改时，48 条指令的空间就太大了。被接受的候选数几乎不变（29–31 个）—— 差别不在”试了多少”，而在”试的是什么”。

Takeaway：in GEPA, the quality of the feedback your evaluator writes is the fuel of optimization. Designing a metric now includes designing its feedback $\mu_f$ — readable compiler errors, which constraint failed, which document was missing. 这和第 14 讲”agent 评估要能定位错误发生在哪一步”是同一个要求：能定位，才能改进。

4.7 Caveats

• 反思需要一个能读懂反馈的 LLM；反馈写得含糊（“incorrect”），反思就退化成盲改（$q \to 0$）。
• 反馈里可能泄露测试信息：如果训练集和测试集有重叠的实体或模板，“expected 1871” 这样的反馈会被写进 prompt，变成背答案。
• GEPA 选择候选靠的是 $D_\text{pareto}$；$D_\text{pareto}$ 小而候选多时，它会过拟合 $D_\text{pareto}$（§7.3）。
• 反过来，GEPA 本身也是一个强大的攻击工具（§7.1）。

5. RL for agents：环境、奖励、数据 · Reinforcement learning for agents

5.1 Why RL, and what an environment is

Trung Vu’s talk starts from a practical difficulty: SFT needs demonstrations, and demonstrations for agents (trajectories with dozens of tool calls) are hard to write from scratch. RL lets an agent improve from outcomes alone. The price is an environment: a simulation of a real-world scenario in which the agent can be stress-tested and trained before deployment.

The five parts of an RL environment (the slides’ “Hello World”: write “Hello World” to /root/hello.txt):

Types of environments: function calling (τ-bench, BFCL: domain tools + an in-process database; the grader checks the database state); search & browsing (Deep Research, BrowseComp, Humanity’s Last Exam: graded with rubrics or reference answers); terminal tasks (Terminal-Bench, SWE-bench: the grader checks the VM state or runs tests); computer use (OSWorld: keyboard and mouse). ARE / Gaia2 (Meta, 2025) from the reading list is a platform for building and evaluating such environments.

5.2 What makes a good environment

The slides list seven criteria; violating each one produces a specific trust failure:

Ways to build environments: by hand (OSWorld, BrowseComp, Terminal-Bench); synthesized from scratch by LLM pipelines (τ-bench, BFCL); converted from existing web data (MLE-Bench); world-breaking (SWE-smith: take a codebase that passes its tests and break it so the tests fail; fixing it is the task); needle-in-a-haystack (BrowseComp: a fact that is hard to find but easy to verify). The last two share one property — they are easy to verify — and a verifiable reward is what makes it safe for RL to optimize hard.

5.3 From rewards to updates: policy gradient, GRPO, DAPO

LLM 作为策略：状态 $s_t$ = 到第 $t$ 个 token 为止看到的所有 token（包括工具的返回），动作 $a_t$ = 下一个 token，奖励只在最后一个 token 上非零（= grader 的分数）。目标是最大化期望回报，策略梯度定理给出

直觉（slides 的说法）：回报为正，就按回报的大小提高这条轨迹上每个 token 的 log 概率；为负就降低。REINFORCE 用整条轨迹的回报当 $\Psi_t$；各种算法的区别在于怎样计算 $\Psi_t$（Schulman et al. 2016 的统一写法）。

GRPO（Shao et al. 2024，DeepSeekMath）对同一个 prompt 采样一组 $G$ 条回答，用组内的均值和标准差做 baseline：

不需要再训练一个 value 网络；再加上 PPO 式的裁剪（防止更新太猛）和对参考模型的 KL。注意一个后果：如果一组的奖励全一样（全对或全错），所有 $A_i = 0$，这组 rollout 不贡献任何梯度。

DAPO（Yu et al. 2025，NeurIPS 2025）在 GRPO 上做了四处修改：Clip-Higher（上下裁剪阈值解耦，防止熵塌缩）、Dynamic Sampling（丢掉奖励全一样的组、补抽新题，让每一步的批次都有信号）、token 级的策略梯度损失、超长回答的奖励塑形；用 Qwen2.5-32B base 在 AIME 2024 上达到 50 分。ScaleRL（“The Art of Scaling Reinforcement Learning Compute for LLMs”，2025）用超过 40 万 GPU 小时的实验拟合了”算力–性能”的 S 形曲线，发现很多设计选择主要改变达到上限的效率，而不是上限本身 —— 这让”用小实验预测大实验”成为可能。

还有一个有争议、但对理解 RL 很有用的观察（Yue et al. 2025, “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?”）：RL 训练出的模型在 pass@1 上更好，但在很大的 $k$ 上，pass@k 不一定超过 base 模型 —— RL 更多是在重新分配模型已有能力上的概率。代码 03 用一个上限（careful 的 logit 最多 +1.0）粗略地表达这一点。

5.4 Agent-specific difficulties

• 环境是新的基础设施：很多环境要起虚拟机 / Docker 容器来执行工具，训练时要同时管理很多个。
• Harness 依赖：同一个模型在不同的 agent 框架（harness）里表现差别很大（slides 用 Terminal-Bench 排行榜说明），可能需要和目标 harness 一起训练。
• 长轨迹：上百次工具调用很快吃光上下文，服务和训练都更贵。
• Credit assignment：做了上千个决定才拿到一个成败信号，每个决定该记多少功？slides 引用 Karpathy 的说法：“sucking supervision through a straw”。这正是 §4.1 GEPA 的出发点的另一面。

5.5 Two case studies from the readings

• Bespoke Labs, “Improving Multi-Turn Tool Use with RL”. Qwen2.5-7B-Instruct, trained with GRPO on a subset of BFCL multi-turn using only 100 training questions: eval accuracy rose from 55% to 78%, with no human or teacher-model demonstrations. Several lessons connect to this lecture: the simplest reward (correctness only) worked best — a three-part reward was unstable, which the authors attribute to reward hacking; runaway completion length (outputs growing into gibberish) was the main problem, and “overlong filtering + a very small KL weight (0.001)” was the most stable fix; updating the reference model every 100 steps helped.
• SWiRL (Goldie et al. 2025, “Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use”): synthesize multi-step reasoning + tool-use trajectories, split each into sub-trajectories (one per action), filter them, then run step-wise RL. Relative accuracy gains over baselines: GSM8K 21.5%, HotPotQA 12.3%, CofCA 14.8%, MuSiQue 11.1%, BeerQA 15.3%; training only on HotPotQA also improved zero-shot GSM8K by a relative 16.9%. The data-filtering finding is worth remembering: filtering by process (is each step reasonable?) worked best, while filtering by outcome (is the final answer correct?) usually hurt — unlike SFT, RL can learn from trajectories that end in a wrong answer; positive and negative examples both help.

（OpenThoughts 和 NaturalThoughts 是选读：两者都在研究”什么样的推理轨迹数据最值得用来蒸馏 / 训练”——又一个”数据筛选决定结果”的例子。）

5.6 In our world: reward hacking is a phase transition

代码 03 把”认真回答一个知识库问题”做成一个小环境。策略在四个动作之间选：careful（认真查，成功率 $\sigma(\text{skill} - d_e)$，RL 能把 skill 最多提高 1.0）、guess（0.2）、abstain（只在无解题上对）、enumerate（列出五个候选，最后再加一句 “or it may be unknown”）。两个 grader：

contains grader 违反了”cheating-proof”和”all tested behaviors specified”：任务要求”给出出生地”，它实际奖励的是”提到出生地”。很多真实的评测脚本就是这样写的（“参考答案是否出现在回答里”）。训练前，策略几乎不列候选（0.047），两个奖励的期望值只差 0.043 —— 漏洞几乎看不见。

GRPO 式训练（每步 8 题 × 8 条 rollout），5 个种子的平均：

这条动态值得慢慢读：

• 前约 8000 次 rollout，两组几乎一模一样。 在 careful 的能力还在提升时，“认真做”本来就是拿 contains 奖励的最好办法。等 careful 的成功率饱和、在很多题上低于 enumerate 的 0.9 时，“列候选”才开始赢 —— 然后共享参数把它从一个题目模板蔓延到所有模板。
• 这是相变，不是缓坡。 5 个种子里，“列候选”过半的时刻在 16000 到 32000 次之间；真实 EM 的峰值 0.71–0.74 出现在 4000 到 12000 次之间。一个短实验会得出”这个 grader 没问题”的结论。
• 差距是早期信号。 每个种子里，proxy − true > 0.05 都和”列候选过半”出现在同一个检查点，或者更早（5 个里有 3 个更早）。算这个差距需要一小份用可验证 grader 打分的审计集。但根本的修复在环境本身：让 grader 防作弊。

5.7 In our world: not too easy, not too hard

代码 03 演示 3 把训练题的难度分布放宽（$d \sim \mathcal N(0, 2.5^2)$）：

• 开始时约 21% 的组没有学习信号；策略变好之后升到 0.41–0.42。“不太容易也不太难”是题目相对于当前策略的属性，会随训练变化 —— 所以 DAPO 在线地做，而不只是训练前筛一次。
• 按 rollout 数比较，两者几乎一模一样（4000 次时都是 0.729）：被丢掉的组照样花了 rollout。按梯度步数比较，dynamic sampling 更快（25 步时 0.663 vs 0.618），代价是多用 rollout（2104 vs 1600）。它换来的是”每一步都有用”，而不是更少的 rollout。我们原本以为它会提高 rollout 层面的样本效率 —— 在这个模拟里没有。

6. 多模态：另一种幻觉，同一种优化 · Multimodal hallucination and preference optimization

6.1 Object hallucination in LVLMs

Large vision-language models (LVLMs) mention objects that are not in the image. Huaxiu Yao’s talk traces this to modality misalignment: the language prior overrides the visual evidence — see a dining table, “say” fork. Standard measures are CHAIR (Rohrbach et al. 2018: the fraction of mentioned objects that are absent from the image, at instance level CHAIR_i and sentence level CHAIR_s) and POPE (Li et al. 2023: binary probes asking “is there an X in the image?”).

LURE (Zhou et al., ICLR 2024) analyzes three statistical factors behind object hallucination: co-occurrence (objects that often appear together with the objects in the image are hallucinated more), uncertainty (objects generated with low probability are more likely to be hallucinations), and position (hallucinations concentrate later in the description). It trains a post-hoc revisor (LVLM Hallucination Revisor) that rewrites suspicious objects; on six open-source LVLMs it reports a 23% improvement on general object-hallucination metrics over the previous best approach.

代码 04 演示 1 用一个极小的”看图说物体”模型重现这三个因素（模型：每一步给每个物体打分 $a\,e(o)\,e^{-ck} + b\cdot\text{prior}(o) - 3.5$，$e$ 是图像证据，$e^{-ck}$ 让越往后越少看图，prior 是场景和共现先验）：

整体 CHAIR_i 0.285、CHAIR_s 0.675：说出的物体里超过四分之一是编的。这三个信号都不需要标准答案就能计算 —— 和第 12、13 讲的无参考幻觉检测、不确定性是同一个思路。

6.2 Where does the preference data come from?

To fix hallucination with preference optimization such as DPO, the key question is where the chosen / rejected pairs come from — labels are expensive. The talk gave two answers:

• POVID (Zhou, Cui, Rafailov, Finn & Yao 2024, “Aligning Modalities in Vision Large Language Models via Preference Fine-tuning”): build preference pairs from SFT data. The SFT ground truth is naturally the chosen answer; rejected answers come from (1) having GPT-4V inject plausible hallucinations into the ground truth (exploiting co-occurrence, a spurious correlation) and (2) adding noise to the image so the model’s own description reveals its inherent hallucination patterns. With about 17k preference pairs (far fewer than the SFT data) on LLaVA-7B, CHAIR, POPE and other metrics improved, and attention to image tokens increased.
• CSR (Zhou et al., “Calibrated Self-Rewarding Vision Language Models”, NeurIPS 2024): without SFT data, let the model generate candidates and score them itself (self-rewarding). The catch: generator and judge are the same misaligned model, so self-scoring carries the same bias. CSR “calibrates” the reward — a step-wise reward = the model’s own language likelihood + an image–response relevance score from an independent model such as CLIP — and runs preference optimization iteratively. The paper reports a 7.62% average improvement across ten benchmarks, and gains over uncalibrated self-rewarding.

One open challenge from the talk: in downstream domains such as medicine, the LVLM itself (a Med-LVLM) is too weak even to serve as a judge — self-improvement presupposes a reliable evaluation signal.

6.3 In our world

代码 04 演示 2：同一个初始模型，三种方式构造偏好对，都用 DPO 训练（600 张训练图；self-rewarding 分 3 轮，每轮 200 张），在 1500 张测试图上评估：

• POVID 式一轮就把 CHAIR_i 降到 0.077，召回率反而从 0.824 升到 0.970（chosen 是”提到所有物体”的标准描述）。DPO 把 $a$ 调高、$b$ 调低：这就是”模态对齐”在参数上的样子。
• 和计划不一样：我们原以为朴素 self-rewarding 会放大偏差，结果它也把 CHAIR_i 降到了 0.020。原因在演示 1：这个模型的幻觉几乎都是低置信度的（≥ 0.3 的提及里一个幻觉也没有），所以”自己的置信度”本身就是很好的幻觉信号。但它付出了代价：长度从 3.67 降到 2.45、召回率从 0.824 降到 0.753 —— 奖励是”每个提及的平均置信度”，最省事的办法就是少说。
• CSR 式的幻觉同样低（0.022），召回率却保持在 0.886：独立的信号奖励”说出图里确实有的东西”，所以不靠少说来降幻觉。

Lesson: judging a hallucination fix by CHAIR alone would have declared naive self-rewarding the winner. Every “reduce the bad thing” metric needs its partner “keep the good thing” metric — CHAIR with recall, groundedness with answer relevance (第 11 讲), refusal rate with helpfulness (第 15 讲), abstention with coverage (第 13 讲). In real LVLMs, hallucinations can be confident when the language prior is strong; then the model’s own confidence stops being a reliable signal, and the independent signal in CSR matters even more.

6.4 Evaluating multimodal RAG

Fall 2023 第 16 讲的第二部分把 RAG triad 推广到多模态（标题就是态度：“Consider multimodal models hallucinatory… unless proven otherwise!”）。例子是一个检索增强的胸片问答系统（X-Ray Insight 项目）：把胸片编码进向量库、附上诊断；用户上传一张片子，检索相似的片子和它们的诊断，再让模型生成诊断。triad 的三条边要改写：

• Context relevance：检索到的图片是否和查询图片足够相似？只看文字查询不够（“Text query is not enough to know if the diagnosis is relevant”），需要图像层面的相关性。
• Groundedness：生成的每个诊断 claim 能否在检索到的报告里找到依据（slides 逐条 claim 打分，例如”心包积液、心影增大”对应报告里的原句）。
• Answer relevance：文字层面的 QA relevance 能覆盖大部分，但多模态的评估器可能更好。

本讲的 §6.1–6.3 和这里是同一个问题的两面：多模态系统生成时容易被语言先验带偏；评估时如果只用文字评估器，也会漏掉”和图像不符”的错误。

7. Goodhart again：优化会”刷”指标 · Optimizers exploit metrics

7.1 Four forms of Goodhart

“When a measure becomes a target, it ceases to be a good measure.” 第 1 讲 §8.3 手动演示过一次；有了优化器，它有四种系统性的形式：

还有一个反过来的形式：优化本身就是攻击。GEPA slides 演示了”把奖励取反”：让 GEPA 去找使模型变差的 prompt。对 AIME 题，GPT-5 Mini 在一个普通 prompt 下得 76%；GEPA 进化出的对抗 prompt 只是在中间插了几句无关的冷知识（蜂蜜不会变质、尼罗河很长、海豚睁一只眼睡觉），准确率就降到 10%。这和第 7、8 讲”评估鲁棒性要用自适应攻击”、第 15 讲的自动化攻击（GCG 式的后缀搜索）是同一回事：你能用来改进系统的优化器，别人也能用来攻击它。

7.2 Optimizing against a biased judge

代码 02 演示 1：同一个 GEPA（小空间），分别对着三种指标优化，各 10 次运行。每格 = judge 分数（测试集模拟）/ 真实 EM；起点（空 prompt）是 0.589 / 0.489：

最终程序里子句出现的次数（每行 10 个程序）：

• 对着 judge 优化：judge 分数从 0.589 涨到 0.852，真实 EM 在起点附近徘徊（0.418–0.519），5 个检查点里有 3 个低于什么都不做。第一个检查点就降到 0.418：优化器最先找到的”改进”就是迎合偏差。
• 它学会的技巧一目了然：10 个程序全都加了 confident 和 elaborate，没有一个保留弃权。更糟的是，judge 的理由（“declined to answer”、“terse”）被反思当成了”梯度” —— 文字反馈让 Goodhart 来得更快、更有方向。
• 对着 EM 优化的程序，judge 只给 0.705：真正更好的系统在有偏的 judge 眼里反而更差。
• 两者平均是一个折中：它找到了”两边都讨好”的写法 —— elaborate（讨好 judge）+ final_tag（保住 EM 的格式）—— 但几乎扔掉了弃权（unanswerable 0.41）。多个指标能挡住最坏的情况，但不能代替一个没有偏差的指标。

7.3 Overfitting the dev set, with a noise floor

代码 02 演示 2：从 200 个随机 prompt 里抽 $N$ 个候选，在一个 $n$ 题的开发集上各采样一次，挑开发集第一。100 次重复的平均：

The null floor runs the same procedure with all $N$ candidates being the same program (nobody is better than anyone) — the “winner” there is simply the one with the luckiest noise. Real candidates differ in true quality and the pick is usually a genuinely good one, so real optimism is smaller than the null floor: the null floor is a conservative upper bound on the optimism, which you can report when you don’t know how different your candidates are (the same idea as the MIA noise floor of 第 1 讲 §5.4 and 第 9 讲).

In practice the situation is often worse than the table: dev sets have tens of items, and every “tweak the prompt and look at the score” is another candidate, so $N$ quietly becomes hundreds. Report numbers from a test set that took no part in any selection — not in prompt search, not in early stopping, not in “which run looked best”.

7.4 Who judges the judge — where you validate matters

代码 02 演示 3(a)：用 200 道带标注的题验证同一个 judge（标注 = EM），但在两个不同程序的输出上：

在优化之前验证一次 judge 是不够的。 优化会把输出推到 judge 最不可靠的地方 —— 这里就是长回答。第 1 讲 §8.3 说要检查 judge 在”被比较的系统之间不同的属性”上的偏差；有了优化器，这句话变成：每一轮优化之后，都要在新的输出分布上重新验证 judge。

演示 3(b) 比较四种估计 EM 的方法（每个候选：300 道无标注题 + 40 道有标注题；分层 Rogan–Gladen 修正的校准集 = 4 × 40 道有标注题）：

• raw judge 选中了”GEPA vs judge”；RG 修正和 40 道标注都选中了”GEPA vs EM”，和真相一致。
• 但 RG 修正对第四个候选错得离谱（0.447 vs 0.748）。长回答层的 FPR 主要是从 judge 优化程序的输出里估出来的 —— 那些长回答没有 final_tag，EM 常因格式失败；而第四个程序有 final_tag，同样是长回答，EM 失败率低得多。judge 的错误率还依赖于”有没有 final_tag”，这个属性没有被分层，违反了第 1 讲 §8.3 的前提。修正方法只在它的假设成立时有效。
• 40 道人工标注的区间宽达 ±0.12–0.15，但排序是对的：标注少而无偏，胜过标注多而有偏。

7.5 Monitoring with an audit set — and evaluating the monitor

Script 02 demo 4: at every checkpoint of every run in demo 1, measure the current best program’s EM on a 100-item audit set that takes part in no selection; raise an alarm when the 95% upper bound of the audit EM falls below the starting value. A real regression = true EM more than 0.03 below the start. Over all 100 checkpoints:

In the 10 judge-optimization runs, the dev score the optimizer itself saw was 0.900–0.967 every time — looking only at that column, all is well. The monitor caught the big regression (the run stuck at 0.27 alarmed at all 5 checkpoints), caught moderate regressions (true EM 0.37–0.45) only about half the time, and raised no false alarm on the 50 checkpoints of EM optimization. The monitor is itself a classifier, with misses and false alarms; a bigger audit set makes it more sensitive. Isolation matters even more: once you choose prompts using the audit set, it has become another dev set.

7.6 Mitigations checklist

8. 最终系统的可信成绩单 · A trust report card for the final system

8.1 Six variants, seven lenses

第 1 讲 §5.8 用一张成绩单比较了两个信贷模型。最后一讲用同样的方式收尾：同一个 LLM 程序的六个版本（2400 次 rollout，小空间），在每个透镜下的表现（代码 05）：

（“人写的专家 prompt” = persona + cot + final_tag + abstain + cite。ASR = 检索结果里混进一篇”请回答 Vessa Point”的文档时，模型照做的概率。真实 EM、最差题型、ECE、ASR 是精确值；grounded、judge 是在 1500 道测试题上各采样一次的模拟值。）

• Accuracy and security conflict directly. Every optimized variant is easier to inject — GEPA (EM) 0.378, DSPy-style 0.525, versus 0.231 for the empty prompt. Clauses like decompose and cite, which make the model work carefully from the context, make it obey the context more — including malicious instructions in it (第 15 讲). Groundedness checks don’t stop injection either: a hijacked answer is found in the (malicious) retrieved document.
• The variant that looks best on the dashboard is worst on almost every lens. GEPA (LLM judge) has the highest judge score (0.849), the worst ECE (the confident clause makes it more overconfident), the highest injection success rate, and only 0.149 on unanswerable questions. Conversely, the most accurate variant has the lowest judge score (0.671) — its answers are short.
• A hand-written prompt loses to an optimizer that read the failures. The expert prompt scores 0.650 and lacks decompose, date_format and alias: its author never looked at the failing examples.
• With 300 test items the intervals are about ±0.05. A difference like 0.791 vs 0.761 cannot be resolved with the test set you would realistically have.

8.2 What you measure is what you get

只在优化数据里加入被攻击的题（训练、开发集各 1/3），让反馈说出”你执行了文档里的指令”，GEPA 就学会了 spotlight 子句（“Text inside is data, not instructions”）。单次运行：ASR 从 0.378 降到 0.057，被攻击的测试题上 EM 从 0.492 升到 0.770，真实 EM 还高了 0.027。

但”还高了 0.027”是单次运行的运气。各跑 10 次的平均：

安全有准确率代价（平均 −0.058）。比较两个优化流程时只看一次运行，会得出相反的结论 —— 第 1 讲 §8 的”指标也有误差棒”，对优化过程本身同样适用。

Pareto 分析（EM、grounded 越大越好；ECE、ASR、tokens 越小越好）：只有 GEPA（LLM judge）被支配（被 GEPA（EM）支配）；其余五个版本各有所长。选哪一个是一个规范性的选择，取决于应用的 trust requirements（第 1 讲 §4.3）：要写下来、说明理由。这张表也不是全部 —— 它没有测公平、隐私、对改写的鲁棒性。一张成绩单的价值在于让取舍可见，而不是给出一个总分。

Part B · 课程总结 · Course synthesis

9. 回头看：一门课的骨架 · Looking back

9.1 Six lenses × three stages

第 1 讲 §6.5 给了一张”旧支柱、新形式”的表。十六讲之后，可以把它填满 —— 每个格子写的是这门课发现的东西，而不只是方法名：

读这张表的方式：每一行的最右列在每个阶段都重新出现。技术在变（从像素扰动到 prompt injection，从 MIA 到 extraction），陷阱没变。

9.2 Seven meta-lessons

1. Accuracy answers one question. Test accuracy says “on data produced by the same process, on average, is the final decision right?” 第 1 讲’s leaderboard winner won through a leaked field; this lecture’s best-looking variant on the dashboard (judge 0.849) is worst on almost every lens. Each lens relaxes one hidden commitment of accuracy.
2. Trade-offs are real, and of three kinds. Some are mathematical (Chouldechova’s identity, L1 §7.1, L5); some appear only past a point (privacy vs utility, “free first, paid later”, L1 §7.2, L10); some can be bought with something else (certified robustness bought with abstention, L1 §7.3, L8; injection resistance bought with a little accuracy, §8.2 here). Knowing which kind you face is most of the work.
3. Measure with uncertainty and a noise floor. Small-group fairness gaps (L1 §8.1, L5), MIA advantage (L1 §5.4, L9), the “best on dev” prompt (§7.3), the comparison of two optimization pipelines (§8.2): any number obtained by taking a max or comparing small samples must be set against what you would get with no signal, and reported with an interval.
4. Assume an adaptive adversary. Robust to a fixed attack is not robust (L7, L8: quantization and distillation “defenses” fall to BPDA and logit attacks); safe against fixed prompts is not safe (L15); the optimizer you use to improve a system can be turned against it (§7.1). Evaluate defenses against attackers who know the defense exists.
5. Certify what you can, abstain on the rest. Certified robustness (L8), differential privacy (L10) and conformal prediction (L13) give guarantees rather than evidence; their shared price is coverage — sometimes the system must say “I don’t know”. This lecture’s unanswerable questions, the judge’s penalty on UNKNOWN, and optimizers deleting the abstention clause show why that price is systematically undervalued by evaluators.
6. Evaluators need evaluation. Sanity checks evaluate attribution methods (L3); judge–human agreement must be stratified by the attributes that vary (L1 §8.3, L11); Agent GPA reports judge–human agreement before using the judge (L14). This lecture adds one more rule: re-validate after every round of optimization, because optimization pushes outputs to where the evaluator is least reliable (§7.4).
7. Report honestly: write the choices down. Model cards and datasheets (L1 §8.4); TPR at low FPR rather than a flattering AUC (L9); ε and the accounting method (L10); the variance over runs, and the results that differed from the plan. Trustworthiness is a property of the report as well as of the system: can a reader tell, from what you wrote, whether the system deserves trust?

9.3 A project methodology: the trust requirements checklist

CS329T 后期的版本是项目制的（Autumn 2025：70% 成绩是项目；“选至少一行应用、两列评估维度”）。Fall 2023 的项目设计讲给了四步：frame → prompt / tune → evaluate → assess reliability。把整门课的教训加进去，就是下面这张清单：

这个循环不是走一遍就结束的：第 6 步发现的问题会回到第 1 步（需求写漏了）或第 3 步（评估器不可靠）。

9.4 Open problems

1. Evaluating what has no ground truth, at scale. 长篇的、开放的、多步的输出越来越多，标注越来越贵；“少量人工标注 + 大量模型标注”（prediction-powered inference）怎样推广到判断会随优化而漂移的场景？
2. Verifiable rewards beyond code and math. RL 最成功的地方都有便宜而可靠的验证器。对于研究报告、医疗建议、商业决策，什么是 cheating-proof 的 grader？
3. Long horizons. Bespoke 的讲座把”人类专家要花几小时 / 几天 / 几个月的任务”列为下一个目标：环境越复杂越难做成可复现的沙盒，反馈越稀疏，credit assignment 越难。
4. Security of self-improving systems. 如果系统会根据反馈改写自己的 prompt 或权重，攻击者能不能通过反馈（或检索内容、工具输出）来”教坏”它？优化器本身成了攻击面。
5. Composing guarantees. 我们有单个性质的保证（DP、certified robustness、conformal coverage），但一个由多个模块、多个工具组成的 agent 的整体保证是什么？
6. Multimodal grounding and evaluation. 当评估器本身（文字 judge、CLIP 分数）也有模态偏差时，怎样评估”和图像一致”？讲座里的 Med-LVLM 例子说明：在最需要可靠性的领域，模型往往弱到不能当自己的 judge。
7. Measuring trust itself. 第 1 讲 §4.1 区分了 trust 和 trustworthiness：目标是让人的信任跟随系统真实的可信度。怎样测量一个人对一个 agent 的信任是否”恰当”，以及我们的报告方式怎样影响它，仍然是开放的人因问题。

10. 小结 · Summary

Six takeaways for Part A:

1. Optimization turns evaluation into the specification. Whatever the metric rewards — including its loopholes — is what you will get.
2. Information per rollout matters most when the search space is large. Reflective methods dominate blind search in the realistic, huge instruction space; in tiny spaces, cheap scalar methods and demonstrations do fine.
3. Keep the stepping stones. Selection that keeps diverse, temporarily worse candidates (Pareto) avoids the stalls of greedy refinement.
4. A good environment is a trustworthy evaluator. Cheating-proof graders, all requirements tested, tasks that are neither trivial nor impossible — violate these and RL will find out, often after a delay.
5. Re-validate evaluators on the outputs the optimizer produces, pair every “less bad” metric with a “still good” metric, and keep an isolated audit set.
6. Trust requirements must be in the objective to be optimized, and their cost must be measured over multiple runs, not one.

11. 常见误区 · Common pitfalls

• “Our optimized prompt scores 0.89 on the dev set.” It was selected on that dev set; its true score in our runs was 0.06–0.10 lower. Report the untouched test set (§4.6, §7.3).
• “The judge agreed with humans 72% of the time before we optimized, so we can trust its scores now.” Optimization moves the outputs to where the judge is weakest; κ fell from 0.440 to 0.085 (§7.4).
• “We combined the judge with EM, so we’re safe from Goodhart.” The average still pulled the program toward the judge’s preferences and dropped abstention (§7.2).
• “Reflection always beats scalar search.” Not in small spaces with cheap demonstrations; its advantage comes from large spaces and informative feedback (§4.6).
• “RL training looked fine for the first 8,000 rollouts, so the reward is fine.” Reward hacking can be a delayed phase transition (§5.6).
• “Dynamic sampling makes RL more sample-efficient.” It makes each gradient step informative; per rollout it was no better here (§5.7).
• “Hallucination dropped, so the model improved.” Check recall/coverage: naive self-rewarding reduced hallucination by saying less (§6.3).
• “Adding the security clause cost nothing — accuracy even went up.” That was one run; over 10 runs it cost 0.058 (§8.2).
• “A grounded answer is a safe answer.” An injected answer is “grounded” in the malicious document (§8.1).

12. 练习 · Exercises

练习 1（信息量与搜索空间）. 一个指令空间有 $M$ 条候选子句，其中只有 $m$ 条有用。(a) 随机提出一个包含 $k$ 条子句的 prompt，它包含某一条特定有用子句的概率是多少？(b) 一次反思若以概率 $q$ 准确指出”该加哪一条”，要以 0.9 的概率至少得到一次正确提案，需要多少次反思？(c) 用 (a)(b) 解释代码 01 演示 3 中：从小空间（$M = 12$）换到大空间（$M = 48$）时，随机搜索从 0.738 掉到 0.634，而 GEPA 只从 0.802 掉到 0.791。

(a) $k/M$（从 $M$ 条里不放回地抽 $k$ 条，某一条被抽中的概率）。$k = 3.5$ 左右时：小空间约 0.29，大空间约 0.07 —— 要同时包含 4–5 条特定子句，概率还要再连乘。(b) $1 - (1-q)^n \ge 0.9 \Rightarrow n \ge \ln 0.1 / \ln(1 - q)$；$q = 0.85$ 时 $n = 2$，与 $M$ 无关。(c) 随机搜索碰到好组合的概率随 $M$ 急剧下降（每条有用子句的概率按 $1/M$ 缩小，几条一起要连乘）；反思的提案由反馈决定，不依赖于空间里有多少无关的句子。只有在反思失败、退回随机修改时（概率 $1 - q$），$M$ 才起作用 —— 所以 $q = 0$ 时大空间里的 GEPA 掉到 0.575（演示 4）。

练习 2（Pareto 选择）. 三个候选在 6 道开发题上的 0/1 分数：A = 111000，B = 110100，C = 110000。(a) 哪些候选在 Pareto front 上？各自的权重（是”最好之一”的题数）是多少？(b) 总是选平均分最高的（贪心）会选谁？(c) 为什么在代码 01 演示 2 中，Pareto front 包含了全部 31 个候选？这对”Pareto 选择 vs 贪心”的区别意味着什么？

(a) 每道题的最高分：题 1、2 都是 1（A、B、C 都是赢家）；题 3 只有 A；题 4 只有 B；题 5、6 没人得分（不计）。C 的答对集合 {1, 2} 是 A 的 {1, 2, 3} 的真子集 → C 被支配，删掉。Front = {A, B}，权重 A = 3、B = 3。(b) A 和 B 平均分相同（0.5），贪心按某个平局规则只选其中一个，另一个永远不会被改进 —— 而 B 恰好会做 A 不会的题 4。(c) 每个候选在每道题上只采样一次、结果有随机性，几乎不可能出现”一个候选答对的题集合严格包含另一个的”，所以没有谁被支配。这时 Pareto 选择退化成”按答对题数加权的随机选择”——它的价值主要来自探索（不总是改进同一个候选），而不是精确的”局部冠军”。用多次采样的平均分（或连续分数）计算 front，可以让它更接近论文的意图，代价是更多的 rollout。

练习 3（GRPO 的零优势组）. 一组 $G$ 条 rollout，每条独立地以概率 $p$ 得到奖励 1。(a) 这一组的所有优势都为 0 的概率是多少？(b) $G = 8$ 时，$p = 0.5$、$0.9$、$0.98$ 各是多少？(c) 用 (b) 解释代码 03 演示 3 中”零优势组的比例”为什么随训练从 0.21 升到 0.41。DAPO 的 dynamic sampling 在 $p$ 很大时要多花多少 rollout 才能凑满一批？

(a) $p^G + (1-p)^G$（全对或全错）。(b) $p = 0.5$：$2 \times 0.5^8 \approx 0.008$；$p = 0.9$：$0.9^8 + 0.1^8 \approx 0.43$；$p = 0.98$：$\approx 0.85$。(c) 训练让 careful 动作在容易的题上的成功率接近 1，于是越来越多的组”全对”；难度分布很宽时，最难的题仍然”全错”。两端都变成零优势。若一组有信号的概率是 $1 - z$，凑满 $B$ 个有信号的组平均要抽 $B/(1-z)$ 组 —— $z = 0.85$ 时约是 6.7 倍的 rollout。这就是为什么 dynamic sampling 换来的是”每步有用”而不是”更省 rollout”，也是为什么”太容易的题”应该被替换成更难的题（课程 / 数据筛选），而不只是被跳过。

练习 4（Goodhart 的方向）. 代码 02 的 judge 是 $P(\text{yes}) = \sigma(-1.1 + 2.9\,c + 0.9\,\ell + 0.6\,\kappa - 1.2\,u)$，其中 $c$ = 内容正确（正确的 UNKNOWN 只得 1.5）、$\ell$ = 长回答、$\kappa$ = 自信语气、$u$ = 回答是 UNKNOWN。(a) 一道无解题，比较两种回答的 judge 通过率：短的、正确的 UNKNOWN vs 长的、自信的、编造的地名。(b) 一道本来能答的题，模型错误地说了 UNKNOWN；如果它改成猜一个（错的）地名并写长、写得自信呢？(c) 由此解释为什么 judge 优化的 10 个程序都删掉了弃权子句。如果你只能修改 judge 的一个系数来阻止这件事，改哪一个？会带来什么新问题？

(a) 正确的 UNKNOWN：$\sigma(-1.1 + 1.5 - 1.2) = \sigma(-0.8) \approx 0.31$；编造的长而自信的回答：$\sigma(-1.1 + 0.9 + 0.6) = \sigma(0.4) \approx 0.60$。judge 更喜欢编造。(b) 错误的 UNKNOWN：$\sigma(-1.1 - 1.2) \approx 0.09$；错的、长而自信的回答 ≈ 0.60。(c) 在 judge 眼里，弃权在两类题上都吃亏，所以删掉弃权、加上 confident / elaborate 永远是”改进”。最直接的修改是把正确 UNKNOWN 的内容分提到和普通正确答案一样（1.5 → 2.9），或去掉 −1.2 的弃权惩罚。新问题：一个不看 KB 的 judge 很难判断”确实查不到”，这样改之后它会奖励过度弃权（本来能答的题也说 UNKNOWN）。更好的办法是给 judge 提供检索到的上下文或参考答案（reference-guided），并在优化后的输出上重新验证（§7.4）。

练习 5（跨讲次：哪个 trade-off？）. 对下面每一个取舍，判断它属于第 1 讲 §7 的哪一类（数学上躲不开 / 只在某个范围之外才出现 / 可以用别的东西换），并给出理由：(a) 本讲 §8 的”准确率 vs 抗注入”；(b) 第 5 讲的”calibration within groups vs equalized odds”；(c) 本讲 §6.3 的”幻觉率 vs 召回率”；(d) 第 10 讲的”ε vs 准确率”。

(a) 主要是”可以用别的东西换”：spotlight 用一点准确率（10 次平均 −0.058）换来 ASR 0.416 → 0.140；它不是数学定理，换一个更好的防御（例如把检索内容和指令在结构上分开）代价可能更小。(b) 数学上躲不开：base rate 不同且分类器不完美时，Kleinberg–Chouldechova 定理保证二者不能同时成立。(c) “只在某个范围之外才出现”：POVID 式同时降低了幻觉、提高了召回（0.285 → 0.077，0.824 → 0.970），说明在初始模型附近两者可以一起改进（模型没对齐时，修好对齐两边都受益）；只有当你靠”少说”来降幻觉（朴素 self-rewarding）时，才在付召回的代价。(d) 第 1 讲 §7.2 的”先免费后付费”：正则化 / 噪声较小时准确率和隐私可以一起改善，超过某一点后 DP 一定要付代价，且代价不均摊（第 10 讲）。

练习 6（为你的项目写一份完整的评估–优化计划）. 选第 1 讲练习 6 的应用之一（医疗笔记结构化，或企业 data agent），按 §9.3 的七步写一份一页计划。必须包含：(a) 2–3 个透镜及其指标，每个”减少坏东西”的指标配一个”保留好东西”的指标；(b) 你会优化什么（指令、示例、权重？），用什么指标和什么文字反馈；(c) 三处防 Goodhart 的设计（例如测试集隔离、审计集、judge 的再验证规则），并说明各自防的是 §7.1 的哪一种；(d) 你会怎样报告”优化后的系统比基线好”（几次运行、什么区间、什么噪声地板）。

(a) Goal fulfillment（SQL 结果与参考答案一致的比例）配 coverage（没有拒答 / 没有返回空结果的比例）；安全（对藏在表注释、工具输出里的注入的 ASR）配 有用性（正常请求的成功率，防止过度拒绝）；groundedness（回答里的每个数字能否在查询结果里找到）配 answer relevance。(b) 优化 planner 和 SQL 生成模块的指令（GEPA 式）；指标 = 执行结果的 EM；文字反馈 = 数据库报错信息、“结果行数不对”、“用错了表 X（应为 Y）”、“执行了注释里的指令”（训练 / 开发数据里混入被攻击的样本，见 §8.2）。(c) ① 一个固定的、从不参与选择的测试集（防开发集过拟合，§7.3）；② 一个 100–200 题的审计集，每轮优化后测 EM，并监控 judge 分数与审计 EM 的差距（防评估器偏差与 reward hacking，§7.5）；③ 每轮优化后，在新程序的输出上用 50 道人工标注重新估计 judge 的分层一致率，κ 下降超过阈值就停止使用该 judge（§7.4）。(d) 基线和优化流程各跑至少 5–10 次（不同的数据划分和种子），报告测试集 EM 的均值 ± 标准差和每次的 Wilson 区间；同时报告”同一流程在随机打乱标签的开发集上能得到的最好开发分数”作为噪声地板；按查询类型拆开报告，并写明哪些类型变差了。

代码相关的练习（“动手改一改”）见 code/lecture16/README.md。

13. 延伸阅读 · Further reading

Prompt / program optimization

• Khattab et al., “DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines”, 2023（ICLR 2024）。
• Opsahl-Ong et al., “Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs”（MIPROv2），EMNLP 2024。
• Yuksekgonul et al., “TextGrad: Automatic ‘Differentiation’ via Text”, 2024；期刊版本 “Optimizing generative AI by backpropagating language model feedback”, Nature 639 (2025)。
• Agrawal et al., “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning”, 2025（以及 Autumn 2025 Oct 28 的 slides；github.com/gepa-ai/gepa）。

RL for agents（Autumn 2025 Oct 23 的阅读材料）

• Bespoke Labs, “Improving Multi-Turn Tool Use with Reinforcement Learning”, 2025（博客）。
• Goldie, Mirhoseini, Zhou, Cai & Manning, “Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use”（SWiRL），2025。
• 环境：Jimenez et al., “SWE-bench”, ICLR 2024；Yang et al., “SWE-smith”, 2025；Xie et al., “OSWorld”, NeurIPS 2024；Terminal-Bench（2025）；Meta, “ARE: Scaling Up Agent Environments and Evaluations”（Gaia2），2025；Yao et al., “τ-bench”, 2024。
• 算法：Shao et al., “DeepSeekMath”（GRPO），2024；Yu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale”, NeurIPS 2025；“The Art of Scaling Reinforcement Learning Compute for LLMs”（ScaleRL），2025；Schulman et al., “High-Dimensional Continuous Control Using Generalized Advantage Estimation”, 2016。
• Yue et al., “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?”, 2025。
• 选读：OpenThoughts（“Data Recipes for Reasoning Models”，2025）；NaturalThoughts（“Selecting and Distilling Reasoning Traces for General Reasoning Tasks”，2025）。

Multimodal

• Zhou et al., “Analyzing and Mitigating Object Hallucination in Large Vision-Language Models”（LURE），ICLR 2024。
• Zhou, Cui, Rafailov, Finn & Yao, “Aligning Modalities in Vision Large Language Models via Preference Fine-tuning”（POVID），2024。
• Zhou et al., “Calibrated Self-Rewarding Vision Language Models”（CSR），NeurIPS 2024。
• Rohrbach et al., “Object Hallucination in Image Captioning”（CHAIR），EMNLP 2018；Li et al., “Evaluating Object Hallucination in Large Vision-Language Models”（POPE），EMNLP 2023。

Goodhart and evaluation

• Manheim & Garrabrant, “Categorizing Variants of Goodhart’s Law”, 2018。
• Krakovna et al., “Specification gaming: the flip side of AI ingenuity”（DeepMind 博客，2020，含一份 specification gaming 的例子清单）。
• Gao, Schulman & Hilton, “Scaling Laws for Reward Model Overoptimization”, ICML 2023。
• Angelopoulos et al., “Prediction-Powered Inference”, Science 2023。

课程总结

• 第 1 讲的延伸阅读（框架、model cards、datasheets）；SYLLABUS.md 里每一讲的”对应原课”。

14. 课程之后 · Where to go next

这是 CS329T 中文讲义的最后一讲。十六讲走过的路：从一个靠泄露字段赢下排行榜的信贷模型（第 1 讲），到解释、公平、鲁棒、隐私四根经典支柱（第 2–10 讲），到 LLM 应用的 RAG triad、grounding、校准（第 11–13 讲），再到会规划、会调用工具、会被注入、会被优化的智能体（第 14–16 讲）。对象一直在变大，问题一直是同一组：它为什么这样决定？对谁有效？输入变了会怎样？泄露了什么？有多确定？说的是真的吗？做的是安全的吗？——以及，我们怎么知道自己的回答是对的？

如果你想继续：

• 构建智能体：CS329A（自我改进的 AI 智能体）和 CS224V（对话式智能体）讲怎样搭建；本课讲怎样评估和保护它们。
• 从人类偏好中学习：CS329H 系统地讲 reward model、RLHF、DPO、LLM-as-a-judge 的偏差 —— 本讲 §6、§7 的理论背景。
• RL 本身：CS234 / CS224R（策略梯度、PPO 的推导、credit assignment）。
• 做一个项目：用 §9.3 的清单，选一行应用、两列透镜。先写下 trust requirements，再搭最简单的系统，在优化之前先验证你的评估器 —— 然后，诚实地报告你发现的一切，包括和你预想的不一样的那些。