SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
- ID: 4d0a16e5
- 原文链接: https://arxiv.org/abs/2605.08366
- PDF: https://arxiv.org/pdf/2605.08366v1
- 作者: Mohit Raghavendra, Soham Dan, Miguel Romero Calvo, Yannis Yiming He, Johannes Baptist Mols, Gautam Anand, Cole McCollum, Edgar Arakelyan, Vijay Bharadwaj, Andrew Park, Jeff Da, MohammadHossein Rezaei, Bing Liu, Brad Kenstler, Yunzhong He
- 日期 / 版本: Submitted on 2026-05-08
- 分类: coding
- 来源类型: paper
- arXiv 分类: cs.LG, cs.SE
- arXiv 备注: 10 pages
- 标签: coding-agent, benchmark, swe-bench, software-engineering, evaluation, 2605-08366
- 质量评分: 4/5
- 抓取时间: 2026-07-27T12:26:02.622727+00:00
中文导读
论文提出 SWE Atlas，目标是 benchmarking coding agents beyond issue resolution它把软件工程 Agent 评测从单一 issue 修复扩展到更广的软件工程任务维度，有助于避免只用 SWE-bench 式成功率代表完整 coding-agent 能力
为什么值得关注
SWE Atlas 试图把 coding-agent 评测从 issue resolution 扩展到更宽的软件工程能力地图
这篇内容值得放进 AAIF，是因为它围绕 Agent / Coding Agent 系统中的一个具体评测或训练问题展开；本页基于 arXiv 元数据、摘要与条目已有摘要整理，未补充摘要之外的实验细节。
关键信息
- 论文标题：SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
- 作者：Mohit Raghavendra, Soham Dan, Miguel Romero Calvo, Yannis Yiming He, Johannes Baptist Mols, Gautam Anand, Cole McCollum, Edgar Arakelyan, Vijay Bharadwaj, Andrew Park, Jeff Da, MohammadHossein Rezaei, Bing Liu, Brad Kenstler, Yunzhong He
- arXiv：https://arxiv.org/abs/2605.08366
- 发布时间 / 修订：Submitted on 2026-05-08
- arXiv 分类：cs.LG, cs.SE
- arXiv 备注：10 pages
- 关联标签：coding-agent, benchmark, swe-bench, software-engineering, evaluation, 2605-08366
English Abstract
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). SWE Atlas differs from prior SWE benchmarks in three key ways: it targets underrepresented but practically important task categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified, agentic task formulations that better reflect real-world usage. Its evaluation framework combines programmatic checks with rubric-based assessment. This goes beyond functional correctness, evaluating software engineering quality, including test and refactor completeness, maintainability, reusable abstractions, and codebase hygiene. We evaluate a range of frontier and open-weight models on SWE Atlas and find that GPT-5.4 and Opus 4.7 achieve the strongest overall performance, while even the best open-weight models score poorly. Our analysis suggests that top models rely on extensive codebase exploration and runtime-driven reasoning. However, even top models consistently struggle with subtle edge cases, complex runtime analysis, and adherence to software engineering best practices. Overall, SWE Atlas provides a complementary evaluation suite for measuring both correctness and engineering quality in coding agents.
English Summary
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). SWE Atlas differs from prior SWE benchmarks in three key ways: it targets underrepresented but practically important task categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified, agentic task formulations that better reflect real-world usage. Its evaluation framework combines programmatic checks with rubric-based assessment. This goes beyond functional correctness, evaluating software engineering quality, including test and refactor completeness, maintainability, reusable abstractions, and codebase hygiene. We evaluate a range of frontier and open-weight models on SWE Atlas and find that GPT-5.4 and Opus 4.7 achieve the strongest o
Obsidian Notes
- 内容获取路径：优先尝试
opencli arxiv paper 2605.08366 -f json，本页使用返回的 arXiv 元数据与摘要回填。 - 中文导读与价值判断均锚定在条目已有摘要、论文摘要、作者、日期与分类信息上。
- 现代站点生成器按
content/{entry.id}.md查找内容页；本文件写入 canonical content 目录，而不是openclaw/content/。