Numerical bugs in softmax and cross-entropy fixed for stable training
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
Researchers have identified and patched two long-standing numerical instabilities in softmax and cross-entropy implementations that affect gradient flow during training. Softmax Collapse causes target gradients to vanish when one logit dominates, breaking the zero-sum property of softmax gradients, while Softmax Cancellation produces spurious gradients during knowledge distillation when student and teacher outputs converge. The fixes address practical training pathologies including the Slingshot phenomenon, accelerate grokking, prevent accuracy collapse, and stabilize distillation. This work matters because these primitives are foundational to nearly all neural network training, making even small numerical corrections potentially impactful across the field.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper isolates two distinct failure modes (Collapse and Cancellation) as separate numerical phenomena rather than treating them as a single stability problem. Critically, it shows that standard numerical safeguards like log-space computation don't prevent Cancellation during distillation, meaning existing implementations carry a blind spot.
This connects directly to the distillation failures documented in 'Capable Teacher, Myopic Scorer' (Oct 20) and 'Right On Q' (Oct 21). Both papers identified why student-teacher training breaks down, but attributed it to architectural or distributional misalignment. This work suggests part of the problem is lower-level: the gradient signal itself becomes corrupted when student and teacher outputs converge. The Slingshot phenomenon mentioned here (rapid accuracy collapse mid-training) also echoes the threshold-like failure modes seen in the ablation study from Oct 20, hinting that architectural choices may interact with these numerical instabilities in ways not yet characterized.
If practitioners report that applying these fixes reduces training instability in knowledge distillation without requiring architectural changes (e.g., temperature scaling or auxiliary losses), that confirms the numerical pathology was a primary blocker rather than a symptom. Watch whether major framework libraries (PyTorch, JAX) adopt these patches in their cross-entropy implementations within the next two quarters; adoption velocity will signal whether the community views this as a critical fix or a niche edge case.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Teacher myopia limits student agent learning in on-policy distillation
Researchers identify a fundamental flaw in on-policy distillation for agent training: teachers score individual tokens without visibility into episode outcomes, causing students to learn locally optimal but globally myopic behaviors. Even when teachers successfully recover from student errors, the student fails because it optimizes for immediate token likelihood rather than task completion. This challenges the…
Mentionssoftmax · cross-entropy · log-softmax · knowledge distillation · grokking
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “A Fix for Softmax and Cross-Entropy”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.