SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch
Source: https://arxiv.org/abs/2607.27167
Authors: Yihao Chen, Shi Chang, Feng Lin, Khaled Chawa, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan
Published: 2026-07-29
Categories: cs.SE, cs.CL
PDF: https://arxiv.org/pdf/2607.27167v1
Abstract
LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances.
Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation.
Inspired by classical requirements engineering, the authors argue that behavioral specification elicitation should be a first-class phase that precedes implementation.
SpecFirst Framework
SpecFirst is a two-stage framework:
1. Specification Agent: Probes the binary and combines observations with documentation into a structured specification 2. Code Synthesis Agent: Uses this specification to drive implementation
This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis.
Results
Evaluated on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability:
- Test pass rates improved by 6.9%–21.3%
- Binary exploration coverage improved by 9.4%–18.5%
- All improvements statistically significant
- Behavioral analysis shows prior specification enables earlier and more sustained code construction
中文概要
SpecFirst 提出在 Agent 从零开始构建程序时，应将行为规格揭示 (behavioral specification elicitation) 作为独立的第一步。传统方法将文档阅读、行为探索和代码合成混在一起，导致 Agent 探测不充分、上下文漂移、早期误解传播到最终实现。SpecFirst 分为两阶段：先由专门的规格 Agent 探测二进制并结合文档生成结构化规格，再由代码合成 Agent 基于规格实现。在 ProgramBench 全部 200 个实例上，测试通过率提升 6.9%–21.3%，二进制探索覆盖率提升 9.4%–18.5%，全部统计显著。