On-policy distillation transfers reasoning but not facts, study finds
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers have isolated a critical asymmetry in on-policy distillation: the technique reliably transfers reasoning skills across novel problem structures but fails to propagate factual knowledge from teacher to student. By systematically varying distillation components across multiple model families, the work reveals that reverse-KL divergence blocks knowledge transfer while forward-KL restores it, and that student rollouts specifically enhance reasoning execution. This finding reshapes how practitioners should configure distillation pipelines depending on whether the goal is capability amplification or knowledge injection, with direct implications for scaling language model training efficiency.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper isolates which divergence measure blocks factual knowledge transfer specifically, not just entropy or collapse risk. Prior work showed reverse-KL reduces entropy and on-policy distillation can fail catastrophically, but this study pins down the mechanism: reverse-KL divergence is the culprit for knowledge propagation failure, while forward-KL restores it without sacrificing reasoning gains.
This directly extends the entropy control framework from the October 2nd paper 'Divergence controls entropy in distillation', which mapped how forward and reverse KL shape student behavior but didn't isolate their effects on knowledge vs. reasoning. The current work operationalizes that theory by showing forward-KL's higher entropy actually enables factual retention, while reverse-KL's convergence pressure erases it. It also contextualizes why the October 2nd 'Gains and Collapse' paper found on-policy distillation produces inconsistent results: the choice of divergence measure determines whether you get capability amplification or knowledge loss, not just whether collapse occurs.
If practitioners report that switching from reverse-KL to forward-KL in production distillation pipelines recovers factual accuracy without degrading reasoning on held-out benchmarks within the next two months, this finding has moved from theory to practice. Conversely, if the accuracy gains don't materialize on real-world knowledge-heavy tasks, the paper's claims about knowledge transfer remain confined to the experimental setup.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.LG
Forward KL divergence inflates student entropy in knowledge distillation
Researchers have mapped how different divergence measures shape student model entropy during knowledge distillation, a foundational technique in LLM training. Forward KL divergence provably increases student entropy above the teacher, while reverse KL reduces it until convergence pressure mounts. This theoretical framework explains why on-policy distillation produces lower-entropy outputs and offers practitioners a principled way…
Mentionson-policy distillation · reverse-KL · forward-KL · language models
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “On-Policy Distillation Teaches New Skills but Not New Knowledge”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.