Distilled models absorb reasoning, not answers, study finds
A new study challenges conventional wisdom about knowledge distillation in large language models by showing that student models absorb teacher reasoning patterns rather than memorizing specific answers. The research systematically tests how well distillation generalizes across domain shifts and multi-teacher scenarios, finding that problem difficulty during training has minimal impact and that even unsolved problems transfer useful reasoning structure. The origin relationship between teacher and student emerges as the critical factor determining transfer success, with same-origin pairs showing strongest alignment. This reframes how practitioners should think about distillation efficiency and suggests reasoning transfer may be more robust than previously assumed.
Modelwire context
ExplainerThe study's core finding is not just that reasoning transfers, but that teacher-student origin alignment (whether they share the same base model lineage) predicts transfer success better than problem difficulty or solution status. This reframes distillation as a structural compatibility problem, not a data quality problem.
This connects to the August gradient bottleneck paper, which also challenged a widely accepted mechanistic claim about LLMs by isolating what actually constrains learning. Both papers use causal or systematic testing to separate confounded factors. The distillation work similarly asks: what's actually limiting transfer? The answer here is architectural kinship, not the reasoning complexity of the training set. This fits a pattern in recent work of questioning whether conventional optimization wisdom holds up under scrutiny.
If practitioners report that distilling a model from a different base family (e.g., Llama teacher to Mistral student) fails to transfer reasoning even on simple problems, while same-family pairs succeed on hard problems, that confirms the origin hypothesis. If origin doesn't matter when you fine-tune the student on the teacher's base model first, that would suggest the barrier is representational rather than structural.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsOn-policy distillation · Large language models
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.