Skip to content
Modelwire
Subscribe

Rotary position distance, not token eviction, limits long-context LLM agents

Source published ·Modelwire updated

Original coverage: arXiv cs.CL ↗·How Modelwire adds context

The development

A new research finding exposes a critical gap in long-context LLM agent systems: keeping needed tokens in the KV cache is insufficient when those tokens drift too far in rotary-position space from the query. Testing on Qwen3-8B with 100K-token coding sessions reveals accuracy collapses from 93% to 64% as definition-to-use distance grows, even when 99.6% of required tokens remain cached. This challenges the core assumption underlying current eviction policies and suggests that positional encoding degradation, not token availability, becomes the bottleneck for extended agent reasoning. The finding has immediate implications for production systems managing million-token sessions on constrained hardware.

Modelwire’s AI-generated summary of coverage from arXiv cs.CL.

Modelwire analysis

Explainer

Our AI-generated reading of the wider context and the next developments to watch.

The paper reveals that current KV cache eviction policies optimize for the wrong metric. Having tokens in cache doesn't matter if rotary position embeddings have degraded too far from the query, making positional distance the actual bottleneck for long-horizon reasoning, not token availability.

This directly challenges the infrastructure optimizations covered in recent coverage. BitDecoding-FP4 and LowRAM both focused on compressing or quantizing the KV cache to fit more tokens in memory, while Intent-Aware Caching and Anthropic's Message Threads optimized eviction and transmission. All assumed that keeping the right tokens cached was the binding constraint. PACE suggests that assumption breaks down past certain context lengths, meaning teams investing in cache efficiency may be solving the wrong problem for extended agent sessions. The finding also connects to the token waste framework from earlier this week, which quantified what gets computed but doesn't matter; PACE identifies what gets computed, stays cached, but still fails because of how the model encodes position.

If production deployments of 100K+ token agent sessions report accuracy recovery after switching from token-count-based eviction to position-aware eviction policies (keeping tokens closer to query position even if fewer total tokens fit), that confirms PACE's diagnosis. If accuracy still collapses despite position-aware caching, the bottleneck is elsewhere and the paper's scope is narrower than claimed.

This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error

Coverage behind this analysis

These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.

  1. ·arXiv cs.CL

    FP4 quantization closes accuracy gap in long-context LLM serving

    Researchers have closed a critical gap in efficient LLM inference by enabling native FP4 quantization for key-value caches without sacrificing accuracy. The work addresses a fundamental tradeoff: existing 4-bit systems either degrade model quality significantly or introduce computational overhead that kills throughput gains. BitDecoding-FP4 uses dual-residual query quantization and underflow-aware scaling to reduce quantization error…

    Read Modelwire coverage →Original source ↗

MentionsQwen3-8B · PACE

MW

How this coverage is produced

Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “PACE: Keeping the Right Tokens in the KV Cache Is Not Enough for Long-Horizon LLM Agents”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.

More to explore

Prefix cache instability leaves LLM agent servers stuck in degraded mode

arXiv cs.LG·

Sampling strategy, not training, drives long-context model failures

arXiv cs.CL·

Pre-training memory operations improves long-horizon agent performance

arXiv cs.CL·
Rotary position distance, not token eviction, limits long-context LLM agents · Modelwire