Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix It
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers have identified a critical failure mode in hybrid linear-attention language models: chain-of-thought fine-tuning systematically destroys long-context retrieval performance, with some models dropping from 67% to 9% accuracy on needle-in-haystack tasks. The root cause lies in CoT training biasing attention gradients toward short-range patterns, corrupting the query-key projections that enable long-range routing. The team proposes QK-Restore, a training-free fix that selectively restores these projections without retraining. This finding matters because it exposes a fundamental tension between reasoning capability and context window utility, two features the industry has been treating as orthogonal improvements.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The finding is architecture-specific in a way the summary understates: hybrid linear-attention models use fundamentally different mechanisms for long-range routing than standard transformers, which is precisely why CoT gradient pressure hits them harder and why a targeted projection-level fix is even possible without full retraining.
This connects directly to the tension exposed in 'Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models' from the same day, where personality conditioning boosted generative tasks while degrading reasoning-heavy ones. Both papers are documenting the same underlying problem from different angles: fine-tuning for one behavioral property systematically degrades another, and prompt-level or inference-time fixes may be the only practical path when retraining is off the table. The PhantomBench coverage also adds relevant pressure here, since a model that loses long-range retrieval under CoT fine-tuning is also a model more likely to confabulate when the relevant context sits far back in the window.
Watch whether QK-Restore holds up when applied to models fine-tuned on longer CoT traces, specifically those exceeding 8K reasoning tokens. If accuracy on needle-in-haystack tasks degrades again at that scale, the fix is addressing a symptom rather than the structural gradient bias.
This interpretation is generated from the summary above and available source metadata. Our methodology · Report an error
MentionsHypeNet · Jet-Nemotron · Chain-of-thought · QK-Restore · Needle-In-A-Haystack
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.