Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
AAIF 摘要
Windowed-MTP 针对百万 token 上下文下 speculative decoding 的 draft KV 成本，给 MTP draft attention 加滑窗与 attention sink，而 target verification 仍保留全注意力。摘要称在 1M 上下文中 draft KV 工作集可降约 99%，SGLang 单卡 decode-step 成本下降 28%–44%。
English abstract
Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap. At million-token context this breaks: an MTP draft head typically runs full attention over the entire KV cache at every draft step, so its read grows linearly with context and comes to dominate the draft cost -- precisely where speculation is most valuable. The effect compounds with draft length (a deep native draft can turn net-negative, slower than no speculation) and sharpens under hybrid/linear-attention targets, where cheaper verification leaves the draft's full-attention read exposed. We apply a StreamingLLM-style sliding window plus attention sink to the draft's attention only (Windowed-MTP), leaving full-attention verification intact. It is training-free, drop-in, and lossless by construction: the full-attention target still decides every accepted token, so windowing changes only which tokens are proposed, never which are accepted. It bounds the draft's KV working set to a constant, dropping ~99% of KV entries at 1M. Across three architecture families (Qwen GDN-MoE 35B/122B and a Mamba2-hybrid NoPE 120B) at 1M context on a single GPU in SGLang, windowing cuts the per-decode-step cost over the shipping native MTP draft by +28% to +44%, an input-invariant margin that widens with context. Since per-token latency is this cost divided by acceptance length, at matched acceptance end-to-end decode latency improves by the same amount, and more where windowing also lifts acceptance, while preserving the target's verified output distribution. Finally, the unread draft KV -- 7.7-11% of total KV at 1M -- is reclaimed via a compact ring buffer at no acceptance or quality cost.
Intake rationale
百万 token 上下文下，draft 模型的 KV 成本已经从小优化变成推理系统瓶颈。
Obsidian evidence excerpt: 这篇处理 million-token context 下 speculative decoding 的 draft KV 成本。做法是只给 MTP draft attention 加滑窗和 attention sink，target verification 仍保留全注意力。摘要称 1M 上下文下 draft KV 工作集可降约 99%，SGLang 单卡实验里 decode-step 成本下降 28%–44%。
Metadata
- Authors: Alagappan Valliappan
- Original date: 2026-07-23
- Source: https://arxiv.org/abs/2607.21535
- Local intake source:
OpenClaw定时任务/论文流水线/2026-07-25-论文流水线.md