Self-distillation critique signals fail to localize reasoning errors
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers challenge a core assumption in on-policy self-distillation for reasoning tasks: that a teacher model scoring student rollouts provides reliable token-level credit signals. Testing against gold error annotations, the work finds that standard distillation objectives fail to localize errors or suppress incorrect tokens, and cannot consistently distinguish successful from failed solutions on identical problems. This finding undermines a promising direction for replacing sparse RL rewards with dense training signals, forcing the field to reconsider how to extract corrective information from model-generated critiques.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper doesn't just show that teacher scoring underperforms; it demonstrates the mechanism of failure: teacher models cannot reliably localize which tokens caused errors, meaning the dense supervision signal that on-policy distillation relies on is fundamentally noisy at the token level, not just sparse.
This directly contradicts the optimism in 'On-Policy Distillation Teaches New Skills but Not New Knowledge' (Oct 7), which found that on-policy distillation successfully transfers reasoning skills. That work assumed the distillation objective was working as intended; this paper reveals the objective itself may be broken for error correction. The finding also reframes 'Sharpen Without Search' (Oct 5), which proposed power distillation to reduce sampling overhead. If teacher scoring cannot reliably distinguish good from bad tokens, then the efficiency gains from single-pass inference may come at the cost of learning from corrupted supervision. Together, these papers suggest the field has been optimizing the wrong bottleneck.
If researchers release ablations showing that replacing teacher token-level scores with gold error annotations recovers the expected distillation gains, that confirms the problem is measurement, not the distillation objective itself. If not, expect a shift toward outcome-only feedback signals (similar to the Caddie approach from Oct 7) rather than token-level credit assignment.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
On-policy distillation transfers reasoning but not facts, study finds
Researchers have isolated a critical asymmetry in on-policy distillation: the technique reliably transfers reasoning skills across novel problem structures but fails to propagate factual knowledge from teacher to student. By systematically varying distillation components across multiple model families, the work reveals that reverse-KL divergence blocks knowledge transfer while forward-KL restores it, and that student rollouts…
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.