ValueDiff improves KV cache eviction for modern attention-normalized LLMs
ValueDiff addresses a fundamental shift in how modern LLMs manage memory constraints. As newer architectures adopt QK-normalization and learned attention sinks to reduce hallucination, they inadvertently weaken the token-importance signals that existing KV cache eviction methods depend on. This paper proposes a geometric alternative: ranking tokens by how far their value vectors deviate from the cache mean, grounded in max-entropy theory. The technique matters because cache efficiency directly impacts inference cost and latency at scale. For practitioners deploying sink-suppressed models under tight memory budgets, this offers a principled replacement for outdated eviction heuristics.
Modelwire context
ExplainerThe paper identifies a specific failure mode: QK-normalization and learned attention sinks, adopted to reduce hallucination, have made older eviction heuristics unreliable by flattening the attention patterns those heuristics depend on. ValueDiff sidesteps this by ranking tokens geometrically rather than by attention weight.
This connects directly to the recurrent state compression work from earlier this week ('What Can a Recurrent State Safely Forget?'), which formalized what information can be safely discarded without breaking task-critical signals. ValueDiff applies that same principle to the KV cache problem: it asks which tokens can be evicted without losing predictive capacity. Both papers treat compression as a problem of identifying orthogonal (task-irrelevant) dimensions. The difference is scope: one works on recurrent hidden states, the other on attention memory. Together they suggest a broader shift toward principled rather than heuristic-driven compression across model components.
If ValueDiff maintains comparable or better perplexity than token-importance baselines when tested on the same sink-suppressed models (RULER or similar) at 4K and 8K context windows, the geometric approach is sound. If performance degrades at longer contexts or on models without QK-normalization, the method may be narrowly tailored to this specific architectural choice rather than generally applicable.
Coverage we drew on
- What Can a Recurrent State Safely Forget? · arXiv cs.LG
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsValueDiff · RULER
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.