Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering

ID: 24c9583a
原文链接: https://arxiv.org/abs/2604.01437
PDF: https://arxiv.org/pdf/2604.01437
作者: Jingyue Li, André Storhaug
日期: 2026-04-01
标签: coding-agents, software-engineering, evaluation, agentic-ai, arxiv
质量评分: 4/5
抓取时间: 2026-07-24T23:34:25+08:00
本地证据: OpenClaw定时任务/论文流水线/2026-07-24-论文流水线.md

中文解读

论文回顾 Agentic AI for Software Engineering 评估中的复现和解释缺口，分析 ICSE/FSE/ASE/ISSTA 相关研究后建议公开 Thought-Action-Result 轨迹与 LLM interaction data。它的价值在于把 coding agent benchmark 从只看结果分数推进到可复查过程。

为什么值得关注

Coding agent 评估下一步要公开过程轨迹，否则 benchmark 分数无法说明系统为什么有效。

English Summary / Abstract

With the advancement of Agentic AI, researchers are increasingly leveraging autonomous agents to address challenges in software engineering (SE). However, the large language models (LLMs) that underpin these agents often function as black boxes, making it difficult to justify the superiority of Agentic AI approaches over baselines. Furthermore, missing information in the evaluation design description frequently renders the reproduction of results infeasible. To synthesize current evaluation practices for Agentic AI in SE, this study analyzes 18 papers on the topic, published or accepted by ICSE 2026, ICSE 2025, FSE 2025, ASE 2025, and ISSTA 2025. The analysis identifies prevailing approaches and their limitations in evaluating Agentic AI for SE, both in current research and potential future studies. To address these shortcomings, this position paper proposes a set of guidelines and recommendations designed to empower reproducible, explainable, and effective evaluations of Agentic AI in software engineering. In particular, we recommend that Agentic AI researchers make their Thought-Action-Result (TAR) trajectories and LLM interaction data, or summarized versions of these artifacts, publicly accessible. Doing so will enable subsequent studies to more effectively analyze the strengths and weaknesses of different Agentic AI approaches. To demonstrate the feasibility of such comparisons, we present a proof-of-concept case study that illustrates how TAR trajectories can support systematic analysis across approaches.

Obsidian evidence excerpt

# 论文流水线 · 2026-07-24
- status: completed
- Obsidian: `/Users/gracker/Library/Mobile Documents/iCloud~md~obsidian/Documents/Obsidian/OpenClaw定时任务/论文流水线/2026-07-24-论文流水线.md`
- Evidence: `/Users/gracker/.hermes/evidence/paper-pipeline/2026-07-24-paper-pipeline-evidence.json`
- route: knowledge `telegram:-1003796963981`
- 检索范围: opencli arxiv recent/search；arXiv 最新批次主要是 2026-07-22 提交。Android 相关内容按 Android 17 / API 37 封顶，未纳入 Android 18 / API 38+。

## 今日精选
1. **Train the Model, Not the Reader**：解释忠实度不能只看重构分数，论文给了审计协议和 RECAP 训练方向，适合做一次精读。
2. **PyroDash**：小模型在 token 级决定是否向大模型求助，给出了成本/准确率两个运行点，适合跟 agent 成本控制放在一起看。
3. **License Laundering in AI Supply Chains**：把 Hugging Face 数据集、模型到 GitHub 应用的许可证传递问题量化到 232270 条链，适合做 AI 工程治理材料。

## 今日论文速报
- **Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations**（Hiskias Dingeto，arXiv:2607.20379，2026-07-22）
  链接: https: