PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference

ID: 19ccf4c0
原文链接: https://arxiv.org/abs/2607.20327
PDF: https://arxiv.org/pdf/2607.20327v1
作者: Niqi Lyu, Pengtao Shi, Wei Qiu, Jianlin Zhong, Sicong Xia, Jianyao Ma, Yicheng Ding
日期: 2026-07-22
分类: cs.CL
标签: llm-inference, small-language-models, routing, cost-optimization, reasoning
质量评分: 4/5
抓取时间: 2026-07-24T20:19:08+08:00

中文解读

PyroDash 关注 SLM/LLM 协作推理的 token-level 交接：小模型在生成中通过控制 token 判断是否把 query 和部分 reasoning trace 交给冻结大模型补全，不需要单独 router重训大模型或访问大模型 logits训练包括控制 token embedding面向 offloading 的 SFT以及带成本项的 GRPO摘要给出两类运行点：=0.05 时平均准确率 64.04%比 LLM-only 高 6.36 个百分点且成本降 20.4%；高成本约束下 LLM token ratio 仅 1.90%，总成本从 49.36 美元降到 1.78 美元

为什么值得关注

它把 SLM/LLM 协作从“外部 router 决策”推进到小模型生成过程内部的 token-level handoff，并把准确率与推理成本放在同一优化目标中；对需要控制 LLM 调用成本的 reasoning 服务有直接参考价值。

English Summary

PyroDash is a token-level SLMLLM collaborative inference framework. During generation, the small model emits a control token to request a single handoff to a frozen LLM, without a separate router, LLM retraining, or LLM logits. Training has three stages: control-token embedding learning, offloading-oriented SFT, and cost-aware GRPO alignment. The abstract reports configurable accuracy/cost points across five math reasoning benchmarks, including 64.04% average accuracy with 20.4% cost reduction at =0.05, and a low-cost setting reducing total cost from USD 49.36 to USD 1.78.

Abstract

Large language models (LLMs) provide strong reasoning capabilities but are expensive to serve at scale, whereas small language models (SLMs) are cheaper but less reliable on difficult problems. We introduce PyroDash, a cost-aware framework for token-level SLM-LLM collaborative inference. During generation, the SLM decides whether to request assistance by emitting a control token. A Collaborate Engine then sends the query and partial reasoning trace to a frozen LLM for completion through a single handoff. The policy is internalized in the SLM, requiring neither a separate router, LLM retraining, nor access to LLM logits. PyroDash trains the SLM in three stages: control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware alignment with Group Relative Policy Optimization. Its reward balances answer accuracy against inference cost normalized by LLM-only inference. Across five mathematical reasoning benchmarks, PyroDash supports different accuracy-cost operating points. With $λ=0.05$, it achieves 64.04 percent average accuracy, 6.36 percentage points above the LLM-only baseline, while reducing cost by 20.4 percent. With $λ=0.6$, it achieves 54.55 percent accuracy with a 1.90 percent LLM token ratio and 0.012 LLM calls per example, reducing total cost from USD 49.36 to USD 1.78. These results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.

Metadata

arXiv ID: 2607.20327
Published: 2026-07-22
Updated: 2026-07-22
Primary category: cs.CL
Categories: cs.CL
Comment: 19 pages, 3 figures
OpenCLI return code: 0