SPADE: Self-Play in Adaptive Synthetic Executable Environments

SPADE：自博弈 RL 框架，单个 LLM 同时扮演两个角色环境设计者以可执行代码写出带 Gym 风格 reset()/step() 接口的完整长时程训练环境，推理 agent 在其中学习用有/无特权提示的奖励差估计 regret，使设计者学会把环境定在 agent 能力边缘但仍可行关键组件：用预训练语料文档为设计者提供 grounding累积环境记忆规模到 30B：八项 held-out 基准较最强固定环境基线平均 +5.3，BFCL-v4 多轮 +5.7，ACEBench-Agent +13.9

SPADE: Self-Play in Adaptive Synthetic Executable Environments

arXiv: 2608.19197
Authors: Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao, Zichen Liu, Junsu Kim, Zijian Zhou, Seungone Kim, Tongzheng Ren, Mickel Liu, Hanfei Yu, Zhaorun Chen, Weiyan Shi, Paul Pu Liang, Luke Zettlemoyer, Yejin Choi, Natasha Jaques
Published: 2026-08-19; categories: cs.CL, cs.AI

Abstract

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.

Why it matters (AAIF scan)

SPADE：自博弈 RL 框架，单个 LLM 同时扮演两个角色——环境设计者以可执行代码写出带 Gym 风格 reset()/step() 接口的完整长时程训练环境，推理 agent 在其中学习。用有/无特权提示的奖励差估计 regret，使设计者学会把环境定在 agent 能力边缘但仍可行。关键组件：用预训练语料文档为设计者提供 grounding、累积环境记忆。规模到 30B：八项 held-out 基准较最强固定环境基线平均 +5.3，BFCL-v4 多轮 +5.7，ACEBench-Agent +13.9。

Source: https://arxiv.org/abs/2608.19197