Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

Dialogue SWE-Bench 把编码智能体评估从全自动修 bug转向通过与用户对话解决真实软件问题，引入 persona-grounded 用户模拟器和对话质量自动评估摘要指出更强的编码模型不一定是更强的对话模型，交互式问题澄清应成为编码智能体独立评估维度

Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

Source: https://arxiv.org/abs/2606.13995

Authors: Brendan King, Jeffrey Flanigan

Published: 2026-06-12

Updated: 2026-06-12

Categories: cs.CL

Comment: 22 pages, 13 figures

PDF: https://arxiv.org/pdf/2606.13995v1

Abstract

AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue SWE-Bench, an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems through dialogue with a user. We design a novel, persona-grounded user simulator to support our task evaluation, and augment our task evaluation with automatic evaluations of dialogue quality. We also propose a new schema-guided agent, aimed at improving the dialogue capabilities of off-the-shelf coding agents, which improves over strong baselines by 3-14%. Our results indicate that better coding models do not always correspond to better dialogue models, suggesting that dialogue capability is a distinct and currently understudied dimension of coding agent performance.

Key Findings

The benchmark reframes coding-agent evaluation from fully autonomous issue solving to dialogue-driven repair of real software engineering tasks.
It introduces a persona-grounded user simulator plus automatic dialogue-quality evaluation, so the agent is judged on clarification and collaboration as well as final task resolution.
A schema-guided agent improves over strong baselines by 3-14%, but the authors also find that stronger coding models are not automatically stronger dialogue models.

中文概要

这篇论文把编码智能体评测从“全自动解 bug”推向“通过对话和用户共同解决真实软件问题”。Dialogue SWE-Bench 使用 persona-grounded 用户模拟器，并加入对话质量的自动评估。摘要里最重要的信号是：更强的编码模型不一定是更强的对话模型，因此交互、澄清和协作能力应该成为编码智能体的独立评估维度。

Why it matters for AAIF

This entry is useful for tracking how agent evaluation is moving beyond single-shot task success: one paper focuses on dialogue-driven coding work, and the other focuses on reward-model assessment for GUI/software-operating agents.

Grounding

Content generated from OpenCLI arXiv metadata/abstract fetched for the canonical arXiv ID.
arXiv URL: https://arxiv.org/abs/2606.13995
PDF: https://arxiv.org/pdf/2606.13995v1