Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

ID: 613e5cf5
Original URL: https://arxiv.org/abs/2608.03893
PDF: https://arxiv.org/pdf/2608.03893v1
Author(s): Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, Mohammad Mahdi Kamani, Ritika Borkar, Makesh Tarun Chandran, Pantea Zardoshti, Bita Darvish Rouhani
Date: 2026-08-04
Category: infra
Source type: paper
Tags: kv-cache, llm-inference, model-routing, cost-optimization
Quality score: 4/5
Fetched at: 2026-08-06T12:26:49+00:00
arXiv categories: cs.LG

中文导读

论文针对生产系统在同一模型家族内切换不同尺寸模型时需要重新 prefill 的成本，提出 cross-model KV cache transfer：在 KV head 数和维度匹配时用闭式线性映射复用源模型 KV cache；摘要报告 Qwen3 14B 到 32B 等场景中可跳过 prefill 并保持质量

为什么值得关注

同族模型路由可以通过 KV cache 转换减少重复 prefill 成本

For infrastructure readers, the paper is relevant because it targets a concrete serving-system cost: model-family routing and mid-conversation switching normally require the receiving model to pay prefill again. The abstract reports a closed-form ridge mapper for matched KV shapes, with Qwen3 14B-to-32B and other family-pair evidence, while also noting failure cases where a nonlinear MLP is needed.

关键信息

Paper title: Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
Authors: Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, Mohammad Mahdi Kamani, Ritika Borkar, Makesh Tarun Chandran, Pantea Zardoshti, Bita Darvish Rouhani
arXiv: https://arxiv.org/abs/2608.03893
PDF: https://arxiv.org/pdf/2608.03893v1
Published: 2026-08-04
Updated: 2026-08-04
Categories: cs.LG
Tags: kv-cache, llm-inference, model-routing, cost-optimization

English Abstract

Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch. We propose cross-model KV cache transfer, where the receiver reuses the source's KV cache, skipping prefill. We find that cross-model KV has substantial linear structure across matched-KV pairs, where source and target share KV head count and per-head dimension. On Qwen3 14B->32B, one source layer explains 56% of variance in the target's keys and 32% in values, rising to 79% and 65% with multiple source layers. Building on this, we design a closed-form ridge mapper that operates per head and proceeds in three steps. First, for each target layer we select the top-k most predictive source layers and concatenate their KV as input. Second, we strip RoPE from the keys before mapping, so the fit is position-free and reusable across context lengths. Third, we fit ridge regression on a small calibration set of 500 FineWeb-Edu sequences of 1,024 tokens each. Surprisingly, across six pairs in three families, this linear mapper retains 73-98% of the receiver's standalone-prefill accuracy on four pairs, while two degrade sharply. A nonlinear MLP recovers up to +37 pp HellaSwag retention on the failures. The mapper runs 2.7-25x faster than re-prefill and remains stable across multi-turn handoff, making cross-model KV cache transfer practical.

English Summary

Obsidian Notes

Content generated from opencli arxiv paper metadata and abstract.
The Chinese guide and relevance assessment are grounded in the abstract and the existing AAIF entry; no paper claims beyond the abstract are added.