Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
- ID: 3deb7451
- 原文链接: https://arxiv.org/abs/2608.20169
- PDF: https://arxiv.org/pdf/2608.20169v1
- 作者: Atsuyuki Miyai, Kiyoharu Aizawa, Toshihiko Yamasaki
- 日期: 2026-08-20
- 分类: agents
- 来源类型: arxiv
- 代码: https://github.com/Agent4Science-UTokyo/Task-CoEvolve
- 质量评分: 4/5
- 抓取时间: 2026-08-24T23:40:00+08:00
中文导读
harness 优化通过迭代改写 agent 框架代码涨分而不动模型权重，但现有做法每轮全量跑固定验证集，大量算力浪费在已失去区分度的任务上。Task-CoEvolve 让验证任务集与 harness 共同演化：按历史结果做方差加权采样，把评估集中在候选 harness 分歧大的任务上，再用采样概率把子集分数还原成全集估计，保证轮次间可比。在在线文本分类与 Terminal-Bench 2.1 上稳定超过固定子集基线，以减少 80% 评估次数匹配全集搜索的最终性能。
Abstract (grounding)
We present a novel approach to efficient LLM agent harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose $\textbf{Task-CoEvolve}$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the agent's capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms fixed-subset baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.
证据摘录（Obsidian 论文流水线 2026-08-24）
任何在跑 agent 迭代评测的人都能直接借鉴：评测预算该花在候选方案分歧处，而不是已经全对或全错的任务上。