Testing Retrieval-Augmented Generation Systems with Chunk Coverage
- source_url: https://arxiv.org/abs/2607.18155
- source_type: paper
- platform: arxiv
- author: Jinhan Kim, Samuele Pasini, Paolo Tonella
- original_date: 2026-07-20
- added_date: 2026-07-22
- category: learning
- tags: rag, testing, chunk-coverage, adequacy, issta, arxiv
- quality_score: 4
- arxiv_id: 2607.18155
- arxiv_categories: cs.SE
- pdf_url: https://arxiv.org/pdf/2607.18155v1
- comment: ISSTA 2026
摘要（中文）
提出 RAG 检索层的测试充分性指标 Chunk Coverage (CC)：衡量测试集至少检索过语料中多少 chunk，不依赖参考答案或人工相关性标注。可指导优先选择能扩大未覆盖检索区域的 query。在临床与金融 RAG 场景中，CC 引导测试达到 50% 可达覆盖率的速度比随机快约 1.7×、比偏冗余策略快约 4.2×；故障检测 APFD 比随机高约 10%–25%。ISSTA 2026。可迁移启发：本地知识库/Obsidian 检索测试应同时统计 chunk 覆盖与回答质量。
Summary (English)
Introduces Chunk Coverage (CC), an oracle-independent test adequacy criterion for the retrieval component of RAG systems. CC measures the fraction of corpus chunks retrieved at least once across a test suite, giving a structural view of exercised retrieval space. CC can guide test selection/generation by prioritizing queries that expand previously unexercised regions. On clinical and financial RAG scenarios, CC-guided testing reaches 50% of attainable coverage 1.7× faster than random and 4.2× faster than redundancy-biased strategies, and improves fault-detection effectiveness (APFD) by 10%–25% over random. Comment: ISSTA 2026.
One-liner
RAG 测试别只问答案对不对：用 Chunk Coverage 量检索空间是否被测到。
Source body / metadata
arXiv abstract grounded intake for 2607.18155. PDF: https://arxiv.org/pdf/2607.18155v1
Introduces Chunk Coverage (CC), an oracle-independent test adequacy criterion for the retrieval component of RAG systems. CC measures the fraction of corpus chunks retrieved at least once across a test suite, giving a structural view of exercised retrieval space. CC can guide test selection/generation by prioritizing queries that expand previously unexercised regions. On clinical and financial RAG scenarios, CC-guided testing reaches 50% of attainable coverage 1.7× faster than random and 4.2× faster than redundancy-biased strategies, and improves fault-detection effectiveness (APFD) by 10%–25% over random. Comment: ISSTA 2026.
Obsidian evidence
- local_note: OpenClaw定时任务/论文流水线/2026-07-22-论文流水线.md
- intake_run: daily-intake-evening 2026-07-22