SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
- ID: 2a3f48ba
- 原文链接: https://arxiv.org/abs/2608.23493
- PDF: https://arxiv.org/pdf/2608.23493v1
- 作者: Jialong Liu, Yuling Shi, Ning Yang, Xiaodong Gu, Zuchao Li
- 日期: 2026-08-24
- 更新: 2026-08-24
- 分类: models
- 来源类型: paper
- 标签: self-reflection, policy-optimization, dense-reward, long-horizon-reasoning, data-efficiency
- 质量评分: 4/5
- 抓取时间: 2026-08-26T15:43:01Z
中文导读
SRPO 把人类学习中自反思式的 credit assignment 内化进 LLM 后训练：模型分析自己跑完的轨迹，把错误合成为简短「反思补丁」，再用反思条件化的教师打分作用在学生的 on-policy rollout 上，得到 token 级稠密训练信号——全程不需要外部 critic、独立奖励模型或更大的教师模型。Qwen3-8B 基座在 AIME'24 拿到 73.3%，只花 scaled SFT 所需训练 FLOPs 的 8%（0.08x）；据论文流水线摘要，WebShop 64.7%、ALFWorld 76.8%、SWE-Bench-Lite 31.2%，数学推理与长程 agent 基准同时刷新。稀疏终端监督换成稠密 token 级信号的数据效率路线又上了一个台阶，代码已开源。
为什么值得关注
不依赖外部 critic 与奖励模型的稠密化路线，把「反思」从推理期技巧变成训练期信号源，数据效率数字（0.08x FLOPs）值得复算。
关键信息
- 论文标题: SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
- 作者: Jialong Liu, Yuling Shi, Ning Yang, Xiaodong Gu, Zuchao Li
- arXiv: https://arxiv.org/abs/2608.23493
- 发布时间: 2026-08-24
- arXiv 分类: cs.AI
- 备注: 无
- 关联标签: self-reflection, policy-optimization, dense-reward, long-horizon-reasoning, data-efficiency
English Abstract
Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at https://github.com/Galleons2029/SRPO
Obsidian Notes
- 元数据与摘要由
opencli arxiv paper 2608.23493 -f json拉取。 - 中文导读锚定在论文摘要声明的 claim 与数字上；未补充摘要之外的实验细节。
- 选题来源: OpenClaw定时任务/论文流水线/2026-08-26-论文流水线.md（评分 4/5 对应流水线 8.x/10 档）。
- 本页由 AAIF daily-intake-evening 任务生成（2026-08-26）。