Value-Aware Stochastic KV Cache Eviction for Reasoning Models
Reasoning models face a hard tradeoff between accuracy and efficiency when handling long chains of thought. This paper identifies why naive KV cache eviction fails: a small set of high-magnitude value states are critical to coherence, and their removal triggers repetitive loops. The authors propose VaSE, a training-free method that combines value-magnitude protection with stochastic sampling to preserve cache diversity. The work matters because it directly addresses the compute bottleneck limiting deployment of reasoning-heavy models like o1, offering a practical path to cheaper inference without sacrificing the extended reasoning that defines their advantage.
Modelwire context
ExplainerThe key insight buried in the framing is that KV cache eviction fails not just because it loses information, but because it loses a specific, identifiable class of information: high-magnitude value vectors that anchor coherence across long reasoning chains. VaSE's stochastic component is also notable because it deliberately preserves randomness in cache selection, which is counterintuitive for a method aimed at reliability.
This paper sits inside a cluster of work Modelwire has been tracking on the practical costs of extended reasoning. The piece on 'Agentic Chain-of-Thought Steering' from June 2nd attacks the same compute problem from the generation side, dynamically controlling how many tokens a model spends reasoning. VaSE attacks it from the memory side, controlling what the model retains while reasoning. Together they represent two complementary pressure points on inference cost. The confidence calibration paper from the same day adds a third dimension: even if you make reasoning cheaper, users may still misread what the model's extended output signals about its reliability.
The real test is whether VaSE's accuracy preservation holds on tasks requiring very long reasoning chains, above 32k tokens, where cache pressure is most severe. If independent replication on AIME 2025 or similar math benchmarks shows degradation at those lengths, the value-protection mechanism may be insufficient without additional training signal.
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsVaSE · KV cache eviction · reasoning models · sparse attention
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.