Score centering fixes training-inference mismatch in LLM reinforcement learning
Training large language models via reinforcement learning faces a fundamental stability problem when inference engines diverge from training setups, a gap known as training-inference mismatch. Researchers have identified drift, a cumulative bias that compounds across training steps, as the primary culprit behind this instability. A new technique called score centering applies a corrective term that neutralizes drift without sacrificing rollout efficiency, the usual tradeoff. Testing across model scales from 600M to 30B parameters shows the method matches or beats importance sampling approaches, with gains widening as mismatch increases. This addresses a critical bottleneck in RL-based LLM alignment and scaling.
Modelwire context
ExplainerScore centering works by applying a corrective term that explicitly targets drift accumulation rather than reweighting trajectories. The key novelty is that it maintains rollout efficiency (the usual cost of bias correction) while still neutralizing the compounding error that importance sampling struggles with at scale.
This connects directly to the distribution shift problem documented in the neural PDE surrogate work from mid-September. Both papers identify how mismatches between training and deployment regimes degrade performance in ways standard transfer learning doesn't catch. Here, the mismatch is temporal (inference diverges from training policy) rather than physical, but the underlying insight is identical: naive approaches fail because the shift compounds. Score centering is a domain-specific fix for RL; the PDE work showed the problem is broader across scientific ML.
If score centering gains hold when tested on held-out policy distributions that the training procedure never saw (not just scaled versions of the same mismatch), that confirms the method generalizes. If the 30B model results degrade relative to smaller scales, that suggests the approach doesn't actually solve drift at frontier model sizes, only masks it.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsLarge language models · Reinforcement learning · Training-inference mismatch · Score centering · Importance sampling
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Score Centering Stabilizes Off-policy Reinforcement Learning”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.