Sequential training beats joint optimization for reasoning LLMs
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers demonstrate that sequencing on-policy distillation before reinforcement learning with verifiable rewards outperforms joint training approaches across reasoning benchmarks. The finding challenges the prevailing strategy of fusing dense token-level supervision with sparse RL signals in a single optimization step. By decomposing the pipeline, the method expands the model's capability space before RL refinement, suggesting that training dynamics and parameter initialization order matter more than previously assumed. This reshapes how teams should architect post-training workflows for reasoning-focused LLMs, with implications for scaling reasoning capabilities efficiently.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The critical finding is not just that sequencing works better, but that it works better because on-policy distillation expands the model's capability space before RL refinement. This suggests the order of operations fundamentally changes what parameter configurations become reachable, not merely that one method is more efficient than another.
This directly extends the Self-Routing work from Alibaba's Qwen team (early September), which showed that adaptive routing of samples to different post-training strategies outperforms uniform recipes. Where Self-Routing routes individual samples dynamically, this paper argues the entire pipeline itself should be sequenced rather than fused. Both challenge the assumption that post-training is a monolithic optimization problem. The on-policy distillation efficiency findings from the single-query paper (also early September) also gain new relevance here: if distillation is cheap enough to run on minimal data, sequencing it before RL becomes computationally feasible for teams with constrained budgets.
If teams report measurable improvements on reasoning benchmarks (MATH, ARC-Challenge, or similar) by adopting sequential pipelines in production models over the next two quarters, this confirms the finding generalizes beyond the paper's test set. If joint training continues to dominate in practice despite this result, it suggests implementation complexity or other hidden costs outweigh the capability gains.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Alibaba's Qwen introduces adaptive routing for post-training optimization
Alibaba's Qwen team introduces Self-Routing, a post-training framework that dynamically routes samples to different optimization strategies based on model behavior rather than applying uniform recipes. By analyzing rollout correctness and confidence signals, the system assigns each sample to GRPO, on-policy distillation, regularization, or skipping without requiring external teachers or extra annotations. Experiments across Qwen3 variants…
MentionsRLVR · on-policy distillation · LLM
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.