CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models

CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models

Source: https://arxiv.org/abs/2609.13060

Author: Blake Olson, Yuhang Song, Emmett McQuinn, Yuan Shangguan

Published: 2026-09-11

Platform: arxiv

中文概要

arXiv 2609.13060（cs.LG）针对扩散语言模型（DLM）在复杂推理与工具调用上落后自回归模型的问题提出 CanvasAnneal：在 RL 早期阶段从更强教师模型注入推理先验以"热启动"扩散画布上的探索，再随训练逐步退火，让模型自行生成更多轨迹。在 MATH500、Countdown、Tau2 上相对 diffu-GRPO 取得提升，且在多个任务上加速奖励增长，但作者也指出收益具有任务依赖性。

English Summary

arXiv 2609.13060, cs.LG. Diffusion Language Models (DLMs) offer parallel generation but lag AR models on reasoning/tool-use. Standard RL hits an exploration bottleneck on DLMs. CanvasAnneal injects reasoning priors from a stronger teacher model during the initial RL phase to warm-start exploration in the diffusion canvas, then gradually anneals guidance off so the model produces more of the trajectory itself. Reports gains over diffu-GRPO on MATH500, Countdown, and Tau2 with task-dependent deltas and faster reward improvement on several tasks.