PAIChecker: 揭示并检查 SWE-Bench 类基准中的 PR-Issue 不对齐

Source: https://arxiv.org/abs/2607.28587

Content fetched: 2026-08-02T12:19:06+08:00

Grounding: opencli arxiv paper

一句话

PAIChecker 发现 SWE-bench Verified 中 13.6% 存在 PR-Issue 不对齐，用三阶段多 Agent 设计达到 92%+ 准确率

关键信息

arXiv ID: 2607.28587
English title: PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks
Authors: Manyi Wang, Junjielong Xu, Pinjia He
Submitted/Published: 2026-07-30
Updated: 2026-07-30
Subjects: cs.SE, cs.AI
PDF: https://arxiv.org/pdf/2607.28587v1
Tags: swe-bench, benchmark-quality, code-agent, pr-issue-alignment, multi-agent
Quality score: 4

中文摘要

PAIChecker 是一个多 Agent 系统，用于检测 SWE-bench 类基准中的 PR-Issue 不对齐问题作者发现 SWE-bench Verified 中 13.6% 的实例存在五种模式的不对齐PAIChecker 采用三阶段设计：模式识别跨 Agent 标签合成代码级验证，在 SWE-Gym 和 SWE-bench Multilingual 上达到 92.12% 和 91.67% 的二分类准确率，超越了基线方法这项研究对代码 Agent 基准有效性具有重要影响

English Summary

PAIChecker is a multi-agent system for detecting PR-Issue misalignment in SWE-bench-like benchmarks. The authors systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. PAIChecker uses a three-phase design combining pattern identification, cross-agent label synthesis, and code-level validation. Experiments on SWE-Gym and SWE-bench Multilingual show it achieves up to 92.12% and 91.67% binary accuracy across four LLM backbones, outperforming baselines.

Why it matters

PAIChecker 发现 SWE-bench Verified 中 13.6% 存在 PR-Issue 不对齐，用三阶段多 Agent 设计达到 92%+ 准确率

arXiv Abstract

SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.

Source Metadata

{
  "id": "2607.28587",
  "title": "PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks",
  "authors": "Manyi Wang, Junjielong Xu, Pinjia He",
  "abstract": "SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.",
  "published": "2026-07-30",
  "updated": "2026-07-30",
  "primary_category": "cs.SE",
  "categories": "cs.SE, cs.AI",
  "comment": "Accepted at the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026)",
  "pdf": "https://arxiv.org/pdf/2607.28587v1",
  "url": "https://arxiv.org/abs/2607.28587"
}