ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
Content-fetcher entry · 2026-10-04 · awesome-ai-field-notes
- URL: https://arxiv.org/abs/2610.02202
- PDF: https://arxiv.org/pdf/2610.02202
- Source: arxiv · Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko, Rulin Shao, Seungone Kim, Graham Neubig, Pang Wei Koh, et al.
- Original Date: Thu, 1 Oct 2026 17:59:47 UTC (4,913 KB)
- Added: 2026-10-04
- Category: models
- Tags: benchmark, retrieval, research-agents, agentic-search, embeddings, scholarly-search
- Quality Score: 5
中文摘要
ScholarCatalyst：衡量检索能启发新研究的论文这一能力的基准184 位近期 CS 论文的第一/lead 作者（覆盖 207 篇论文）通过自动化流水线标注哪些候选文献实际或可能推进了自己的项目，并给出详细理由；任务设定为在项目开始时点可用的文献范围内，给定初始研究问题检索这些论文结果：agentic search（把同一个 embedding 检索器当工具调用）Recall@20 0.42，反而不如纯 embedding 检索的 0.48；即便是训练数据里可能见过这些已完成论文的 Claude Fable 5.1 agent 也只有 0.51说明科学家式嗅探哪篇前作关键仍是 AI 系统的明显短板2026-10-01 提交，cs.AI/cs.CL/cs.IR
English Abstract
What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.
One-liner
ScholarCatalyst：衡量检索能启发新研究的论文这一能力的基准184 位近期 CS 论文的第一/lead 作者（覆盖 207 篇论文）通过自动化流水线标注哪些候选文献实际或可能推进了自己的项目，并给出详细理由.
注：本文件为 content-fetcher cron 回填；opencli arxiv 拉取失败（COMMAND_EXEC），改以 opencli web read 抓取 arXiv abs 页面，仅依据可见的标题/作者/提交时间/分类/摘要生成，未补充摘要之外的实验细节。