Language models hide rather than erase old knowledge during finetuning
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
Researchers have identified why language models appear to forget old knowledge during finetuning, yet often retain it in recoverable form. The work reveals that finetuning shifts representations along shared directions rather than erasing them, and that normalization mechanisms can restore access to buried facts once new training stabilizes. This mechanistic insight matters for practitioners: it suggests forgetting during adaptation is often reversible, opening paths to safer continual learning without catastrophic knowledge loss. The finding bridges interpretability and practical finetuning, clarifying when model updates preserve rather than destroy learned structure.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper's core contribution is not that forgetting happens during finetuning (known), but that it's representational drift rather than weight erasure. The recoverable nature of buried facts under normalization is the mechanistic detail that flips the risk calculus.
This directly addresses the catastrophic forgetting problem that has dominated recent coverage. The off-policy merging work from October 5th showed that on-policy constraints may be overfit solutions, and the SFT sampling paper from October 1st challenged whether forgetting during adaptation is as severe as assumed. This paper provides the mechanistic explanation for why those empirical results hold: knowledge isn't destroyed, it's occluded. The stratified retention framework from October 2nd also becomes more actionable here, since practitioners now know that 'forgotten' knowledge might be recoverable without architectural redesign.
If practitioners report that applying post-hoc normalization to finetuned models recovers performance on held-out pretraining benchmarks without retraining, that validates the reversibility claim. If this doesn't replicate across model scales (7B to 70B+) or across different normalization schemes (LayerNorm, RMSNorm, etc.), the finding may be narrower than presented.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.LG
Off-policy merging outperforms on-policy distillation for continual learning
Researchers challenge the prevailing assumption that on-policy training is essential for continual learning in post-trained models. The work demonstrates that off-policy merging outperforms on-policy self-distillation, a technique designed to mitigate catastrophic forgetting when adapting models to new data. This finding matters because real-world knowledge updates are often off-policy, yet current best practices force conversion to…
MentionsLanguage models · Finetuning · Associative memory
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.