SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents
Source: https://arxiv.org/abs/2609.04167
Authors: Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen, Hongyu Zhang, Guanbin Li
Published: 2026-09-03
Categories: cs.SE, cs.AI
PDF: https://arxiv.org/pdf/2609.04167
Abstract
Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development. We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance. We construct SWE-Gate with 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains. Experiments with four LLM backends spanning different capability levels under a common coding-agent scaffold reveal a substantial gap between functional success and success under the complete repair specification: among 644 repairs that pass the functional tests, 221 fail to satisfy the provided review constraints. These findings show that functional-only evaluation overestimates agents' ability to satisfy the full requirements of repository-level repair tasks. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/SWE-Gate.
Key Points
- Existing repository-level SE benchmarks primarily measure whether generated patches pass functional tests, overlooking review-derived acceptance constraints that often decide whether a patch is acceptable in real-world development.
- SWE-Gate explicitly evaluates review constraint compliance alongside functional correctness: constraints derive from real pull-request review comments, and repair instances are synthesized around them.
- Each instance provides separate functional and constraint tests together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance.
- The benchmark comprises 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains.
- With four LLM backends spanning different capability levels under a common coding-agent scaffold, 221 of 644 repairs that pass the functional tests fail the provided review constraints - functional-only evaluation overestimates agents' ability to satisfy the full repair specification.
- Replication package (code, data, experimental results) is open-sourced at github.com/DeepSoftwareAnalytics/SWE-Gate.
中文概要
SWE-Gate：在功能测试之外显式评测 code review 约束遵从的仓库级 benchmark约束取自真实 PR review 评论并围绕其合成修复实例；每个实例提供功能测试约束测试不合规补丁与 gold 补丁，把问题解决能力与 review 约束遵从拆开度量共 303 个实例，覆盖 75 个开源 Python 仓库四种能力梯度的 LLM 后端在同一 agent scaffold 下实验：644 个通过功能测试的修复中 221 个不满足 review 约束，仅测功能会高估 agent 完成仓库级修复完整要求的能力复现包（代码数据实验结果）已在 GitHub 开源2026-09-03 提交 cs.SE