Linear KV cache transfer enables cost-free model switching in production
Researchers have identified a linear structure in key-value caches across model families, enabling efficient cache reuse when switching between different-sized LLMs without recomputing the prefill. Using ridge regression mappers applied per attention head, the technique recovers 56-79% of variance in keys and 32-65% in values when transferring from Qwen3 14B to 32B, substantially reducing latency and compute during model cascading and mid-conversation routing. This addresses a real production bottleneck where cost-quality tradeoffs currently force full recomputation, making it directly applicable to inference optimization in multi-model deployments.
Modelwire context
ExplainerThe paper's key insight is that KV cache structure remains linearly predictable across model sizes within a family, not just within a single model. This means the bottleneck isn't the cache itself but the assumption that you must recompute it when routing between different-capacity models.
This directly addresses the inference optimization layer that Baseten's team outlined last month (The Inference Frontier). While that coverage emphasized disaggregated prefill/decode pipelines and kernel rewrites as the path to 10x throughput, this work targets a specific sub-problem: the cost of prefill reuse during model cascading. The variance recovery rates (56-79% for keys, 32-65% for values) suggest the mapping isn't perfect, but it's good enough to avoid full recomputation in multi-model deployments where latency and cost tradeoffs currently force routing decisions.
If Baseten, Anyscale, or another inference platform announces support for cross-model KV cache transfer in their routing layer within the next two quarters, that signals production viability. If the technique remains confined to research without vendor adoption by Q1 2027, the variance recovery may be too lossy for real workloads where output quality matters.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsQwen3 · KV cache · ridge regression
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.