ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?
- ID: f49d10d6
- Original URL: https://arxiv.org/abs/2608.03874
- PDF: https://arxiv.org/pdf/2608.03874v1
- Author(s): Tianyi Guan, Yiding Wang, Haotong Yang, Siyuan Cao, Shirui Liu, Yi Hu, Jiaqi Li, Muhan Zhang
- Date: 2026-08-04
- Category: agents
- Source type: paper
- Tags: agent-evaluation, skill-library, continual-learning, benchmarks
- Quality score: 4/5
- Fetched at: 2026-08-06T15:43:42+00:00
- Obsidian evidence: OpenClaw定时任务/论文流水线/2026-08-06-论文流水线.md
中文导读
ContinualSkillBench 评测带外部 skill library 的 LLM Agent 能否在连续任务中演化能力。基准覆盖 5 个领域，每个领域 100 个按难度递增、存在跨任务技能复用机会的 subtasks。作者发现顺序执行通常能提升表现，但收益随模型和领域波动；显式 skill 维护平均上未必明显超过 in-context 适应，但在可复用流程和精确输出任务上有选择性收益。
为什么值得关注
为长期运行 agent、技能库演化和回归评估提供了实验框架。
English Summary
Modern agent frameworks equip large language models with external skill libraries to solve complex tasks. However, it remains unclear whether these systems can effectively evolve their skills and whether the resulting skills improve task-solving capabilities. To bridge this gap, we introduce ContinualSkillBench, a dynamic evaluation framework for in-context continual skill learning. It covers five representative domains, each containing 100 interconnected subtasks ordered by increasing difficulty and opportunities for cross-task skill reuse. Our experiments show that sequential execution generally improves performance, but the gains vary substantially across models and domains. Moreover, in-context learning performs comparably to explicit skill maintenance on average, suggesting that much of the improvement arises from adaptation to prior context and feedback rather than reusable skill abstraction alone. Explicit skills nevertheless provide selective benefits for tasks requiring reusable procedures or precise outputs. We further find that less capable models tend to accumulate larger, more fragmented collections of task-specific skills. These findings show that current in-context skill evolution mechanisms can support continual adaptation, but still struggle to consistently consolidate experience into robust and transferable skills.
原文摘要 / Source Excerpt
ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?
- arXiv: https://arxiv.org/abs/2608.03874
- PDF: https://arxiv.org/pdf/2608.03874v1
- Authors: Tianyi Guan, Yiding Wang, Haotong Yang, Siyuan Cao, Shirui Liu, Yi Hu, Jiaqi Li, Muhan Zhang
- Published: 2026-08-04
- Updated: 2026-08-04
- Categories: cs.AI, cs.CL, cs.LG
Abstract
Modern agent frameworks equip large language models with external skill libraries to solve complex tasks. However, it remains unclear whether these systems can effectively evolve their skills and whether the resulting skills improve task-solving capabilities. To bridge this gap, we introduce ContinualSkillBench, a dynamic evaluation framework for in-context continual skill learning. It covers five representative domains, each containing 100 interconnected subtasks ordered by increasing difficulty and opportunities for cross-task skill reuse. Our experiments show that sequential execution generally improves performance, but the gains vary substantially across models and domains. Moreover, in-context learning performs comparably to explicit skill maintenance on average, suggesting that much of the improvement arises from adaptation to prior context and feedback rather than reusable skill abstraction alone. Explicit skills nevertheless provide selective benefits for tasks requiring reusable procedures or precise outputs. We further find that less capable models tend to accumulate larger, more fragmented collections of task-specific skills. These findings show that current in-context skill evolution mechanisms can support continual adaptation, but still struggle to consistently consolidate experience into robust and transferable skills.