Sequential training beats joint optimization for reasoning LLMs
Researchers demonstrate that sequencing on-policy distillation before reinforcement learning with verifiable rewards outperforms joint training approaches across reasoning benchmarks. The finding challenges the prevailing strategy of fusing dense token-level supervision with sparse RL signals in a single optimization step. By decomposing the pipeline, the method expands the model's capability space before RL refinement, suggesting that training dynamics and parameter initialization order matter more than previously assumed. This reshapes how teams should architect post-training workflows for reasoning-focused LLMs, with implications for scaling reasoning capabilities efficiently.
Modelwire context
ExplainerThe critical finding is not just that sequencing works better, but that it works better because on-policy distillation expands the model's capability space before RL refinement. This suggests the order of operations fundamentally changes what parameter configurations become reachable, not merely that one method is more efficient than another.
This directly extends the Self-Routing work from Alibaba's Qwen team (early September), which showed that adaptive routing of samples to different post-training strategies outperforms uniform recipes. Where Self-Routing routes individual samples dynamically, this paper argues the entire pipeline itself should be sequenced rather than fused. Both challenge the assumption that post-training is a monolithic optimization problem. The on-policy distillation efficiency findings from the single-query paper (also early September) also gain new relevance here: if distillation is cheap enough to run on minimal data, sequencing it before RL becomes computationally feasible for teams with constrained budgets.
If teams report measurable improvements on reasoning benchmarks (MATH, ARC-Challenge, or similar) by adopting sequential pipelines in production models over the next two quarters, this confirms the finding generalizes beyond the paper's test set. If joint training continues to dominate in practice despite this result, it suggests implementation complexity or other hidden costs outweigh the capability gains.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsRLVR · on-policy distillation · LLM
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.