JetSpec Enables Up to 9.64x Lossless LLM Inference Speedup with Up to 1000TPS
备注：以下为自动提取的正文内容。
Side-by-side comparison of decoding speed among JetSpec, DFlash and AR baseline.
Low drafting cost: generate many tree nodes in one draft-head forward pass.
High acceptance: condition every node on its branch prefix, not just on its absolute future position.
Lossless verification: let the frozen target verify the tree and commit only the prefix it agrees with.
Generalizability: largest gains appear on reasoning-heavy math and coding tasks, in consistent with our training data choice. JetSpec also generalizes with >4x speedup on open-ended conversational tasks.
Non-greedy sampling: gains shrink but remain consistent, showing the causal-tree benefit is not limited to deterministic decoding.
Budget scaling: larger tree budgets help JetSpec more reliably because the draft tree stays branch-conditioned.
In this experiment we compare tree quality with and without causality enforced. The gap is measured the drafter’s log-probability difference, in nats (natural-log units), between its top-ranked branch and the target’s preferred continuation. A small gap means the tree contains the branch the target is more likely to accept and therefore a higher quality. On MATH-500, without loss weighting the block-diffusion head is miscalibrated, accepting a mean 4.84 tokens per round against the causal head’s 9.46.
Website and Demos: https://jetspec-project.github.io/jetspec-web/