Quantized LLMs recover reasoning via student-generated training paths
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers tackle a fundamental problem in model compression: quantized LLMs lose reasoning ability on long-horizon tasks because training uses fixed teacher trajectories that diverge from student generation paths. The paper identifies distributional misalignment in next-token prediction as a core failure mode and proposes self-generated trajectories to close the gap. This addresses a practical bottleneck for deploying reasoning models on edge hardware, where low-bit quantization is essential but current methods fail to preserve chain-of-thought capabilities.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper isolates a specific failure mode: quantized student models diverge from teacher trajectories during inference, causing next-token distributions to shift in ways training never saw. This isn't just performance loss; it's a mismatch between what the model learned and what it actually does at runtime.
This connects directly to the October 17th finding that RL-trained models make fundamentally different token choices at only 4% of positions yet capture 90% of reasoning gains. Here, the problem reverses: quantization forces widespread distributional shifts that destroy reasoning even when individual token choices might seem preserved. The October 21st work on chain-of-thought shortcuts adds another layer, suggesting that when quantized models lose reasoning, they may fall back harder on statistical patterns rather than genuine rationales. Together, these papers sketch a picture where reasoning robustness depends on maintaining alignment between training and inference distributions, whether through RL-guided decision points or through careful trajectory matching during compression.
If the self-generated trajectory method preserves reasoning performance on long-horizon math benchmarks (MATH-500, Aime24) at 4-bit quantization with less than 2% accuracy drop compared to full precision, the approach is viable for edge deployment. If performance degrades beyond 5%, it suggests the distributional mismatch is deeper than next-token prediction alone.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
RL's math gains come from 4% of token choices, study finds
A new analysis of reinforcement learning's impact on reasoning reveals that RL-trained models make fundamentally different token choices than base models in only 4% of positions, yet these divergences account for 90% of performance gains on mathematical problems. The finding suggests RL doesn't broadly reshape reasoning but rather teaches models to recognize and exploit specific…
MentionsQuantization-aware training · Large language models · Low-bit quantization
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.