Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix It
Researchers have identified a critical failure mode in hybrid linear-attention language models: chain-of-thought fine-tuning systematically destroys long-context retrieval performance, with some models dropping from 67% to 9% accuracy on needle-in-haystack tasks. The root cause lies in CoT training biasing attention gradients toward short-range patterns, corrupting the query-key projections that enable long-range routing. The team proposes QK-Restore, a training-free fix that selectively restores these projections without retraining. This finding matters because it exposes a fundamental tension between reasoning capability and context window utility, two features the industry has been treating as orthogonal improvements.
Modelwire context
ExplainerThe finding is architecture-specific in a way the summary understates: hybrid linear-attention models use fundamentally different mechanisms for long-range routing than standard transformers, which is precisely why CoT gradient pressure hits them harder and why a targeted projection-level fix is even possible without full retraining.
This connects directly to the tension exposed in 'Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models' from the same day, where personality conditioning boosted generative tasks while degrading reasoning-heavy ones. Both papers are documenting the same underlying problem from different angles: fine-tuning for one behavioral property systematically degrades another, and prompt-level or inference-time fixes may be the only practical path when retraining is off the table. The PhantomBench coverage also adds relevant pressure here, since a model that loses long-range retrieval under CoT fine-tuning is also a model more likely to confabulate when the relevant context sits far back in the window.
Watch whether QK-Restore holds up when applied to models fine-tuned on longer CoT traces, specifically those exceeding 8K reasoning tokens. If accuracy on needle-in-haystack tasks degrades again at that scale, the fix is addressing a symptom rather than the structural gradient bias.
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsHypeNet · Jet-Nemotron · Chain-of-thought · QK-Restore · Needle-In-A-Haystack
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.