LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recove...

临床 AI scribe 的主要错误是遗漏（记下却没写进病历），而标准检查手段是再请一个 LLM 当 judge这篇用 500 对单错误笔记（含 298 条"事实确实缺失"与 202 条"添加/篡改"对照）系统测了八种 judge 设计：配对判别对添加/篡改达 0.79-0.94，对遗漏只有 0.50-0.63（接近抛硬币）；单笔记场景下没有任何 judge 设计能把遗漏标记率做到显著高于完美笔记；改措辞投票prompt 优化都只是移动工作点恢复检测靠任务重构先把 transcript 事实列出来再逐条对照，per-fact 流水线在 2.7% 误报率下点名缺失事实，单次调用版检测率更高且成本只有 1/10结论远超临床场景：LLM judge 验证存在性不验证缺失性，做代码 review文档审计合规检查的人都该把缺失检查显式做成枚举-对照任务

LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It

ID: 61cb7313
原文链接: https://arxiv.org/abs/2608.31016
PDF: https://arxiv.org/pdf/2608.31016v1
作者: Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris
日期: 2026-08-31
抓取时间: 2026-09-02T15:31:00Z
分类: agents
来源类型: paper
语言: zh
标签: llm-judge, evaluation, clinical-ai, omission, audit
质量评分: 5/5

中文导读

一句话点评

临床 AI scribe 的主要错误是遗漏（记下却没写进病历），而标准检查手段是再请一个 LLM 当 judge这篇用 500 对单错误笔记（含 298 条"事实确实缺失"与 202 条"添加/篡改"对照）系统测了八种 judge 设计：配对判别对添加/篡改达 0.

English Abstract / Summary

Across 500 single-error note pairs (298 omission, 202 added-or-altered) the authors show judge discrimination reaches 0.79-0.94 for additions/alterations but only 0.50-0.63 for omissions; on single notes no judge design flags omissions significantly above the perfect-note baseline, and rephrasing, voting, and prompt tweaks only shift the operating point. Recovery requires task restructuring: enumerate facts from the transcript first, then check each one. The per-fact pipeline reaches 36.9% single-call detection at 2.7% false-positive rate, ten times cheaper than the single-call baseline; doctors sided 10/10 with the pipeline on disagreements.

Obsidian Notes

由 daily-intake-evening 2026-09-02 cron 从当日 Obsidian 摘要（论文流水线 / AK-RSS / ClawFeed / X 书签消化）发现并入库存量阶段。
中文导读与判断均锚定在条目已有摘要、源页面正文、作者、日期与分类信息；未补充源页面之外的实验细节。