HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
- ID: a02c8e7a
- arXiv: https://arxiv.org/abs/2607.25398
- PDF: https://arxiv.org/pdf/2607.25398
- 作者: Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta, Edwin Chen
- 发布: 2026-07-28
- arXiv 分类: cs.AI, cs.CL
- Comment: Accepted to the Workshop on Agent Behavior (WAB) at COLM 2026
- 抓取时间: 2026-07-30
中文导读
HANDBOOK.md 把「把公司手册/CLAUDE.md/政策塞进 context 就能约束 agent」这个部署假设拿来打脸。65 个企业任务、20-124 页 SOP、824 条程序化判分，最强配置严格通过率也只有 36.2%。失败模式稳定：近端请求压过站立政策、做完检查却按反结果行动、长程丢规则细节。
为什么值得关注
把手册塞进 context 就能约束 agent？最强配置严格通过率也只有 36.2%
English Abstract
Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document actually constrains its behavior over an extended tool-use horizon. We present HANDBOOK.md, a benchmark of 65 agentic tasks modeled on how enterprise employees follow company handbooks. Each task places an agent in a self-contained company environment, a file workspace together with mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol, and instructs it to carry out routine professional work governed by an expert-written standard operating procedure of 20 to 124 pages. Tasks span five domains (finance, medical billing, insurance, logistics, and HR) and ten fictional companies. To resist memorization, every task modifies one of ten base handbooks, altering the specific rules and thresholds on which grading turns, so no two tasks share a policy. Grading is fully deterministic: each task carries a rubric of programmatic criteria (824 in total) that check both that required actions occurred and that prohibited actions did not. Under strict grading, where a trial passes only if every criterion is satisfied, the best of thirty evaluated model configurations passes 36.2% of trials, and most frontier configurations remain below 25%. Failures follow consistent patterns: agents let a plausible in-environment request override the standing policy, perform a required check and then act against its result, lose rule details over long horizons, and report compliance they did not achieve. We release all tasks, environments, and the evaluation harness.