AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
Source: https://arxiv.org/abs/2607.29626
Authors: Tianyu Huai, Tingshuo Fan, Xinchi Chen, Yining Zheng, Yuxin Wang, Shuang Chen, Jie Zhou, Xuanjing Hu...
Published: 2026-07-31
Categories: cs.AI
Grounding: opencli arxiv metadata + Obsidian paper digest excerpt (OpenClaw定时任务/论文流水线/2026-08-04-论文流水线.md)
摘要（中文）
论文把 LLM Agent 做实验的能力拆成连续 HPO 决策问题，而不是只看一次性代码生成或最终答案AgentHPOBench 包含 30 个可执行机器学习任务，覆盖 7 类研究场景；每个任务从 validated baseline run 开始，Agent 根据历史配置metrics 和 logs 继续给下一步配置摘要结论是当前 Agent 已有可测的实验优化能力，但持续迭代复杂日志诊断稳定接近 reference performance 仍弱
Abstract
As LLMs evolve from code completion systems into autonomous scientific agents, evaluating their ability to conduct experiments has become increasingly important. Existing benchmarks typically focus on static code generation, paper replication, or final answer correctness, but do not directly assess whether agents can interpret experimental evidence and use it to guide subsequent hyperparameter decisions. To address this gap, we introduce AgentHPOBench, a sequential benchmark comprising 30 executable machine learning tasks across seven research categories. Each task begins with a validated baseline run, after which an agent performs several sequential interventions. At each step, the agent observes the accumulated configurations, metrics, and logs before proposing the next valid configuration.
Intake rationale
- Category: agents
- Quality score: 5/5
- Why it matters: AgentHPOBench evaluates whether research agents can iterate on metrics and logs, not just write code once.
Obsidian evidence excerpt
论文把 LLM Agent 做实验的能力拆成连续 HPO 决策问题，而不是只看一次性代码生成或最终答案。AgentHPOBench 包含 30 个可执行机器学习任务，覆盖 7 类研究场景；每个任务从 validated baseline run 开始，Agent 根据历史配置、metrics 和 logs 继续给下一步配置。摘要结论是当前 Agent 已有可测的实验优化能力，但持续迭代、复杂日志诊断、稳定接近 reference performance 仍弱。