Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data

The problem isn't that RL is failing to improve weak models — it's that RL is failing precisely because the base model is already too good. Saturation at the data level, not the algorithm level, is what collapses the training signal, which means benchmark choice is now a first-order design decision, not just an evaluation afterthought.

This connects directly to 'When Can LLMs Learn to Reason with Weak Supervision?' from the same day, which found that models generalizing well show prolonged phases where reward and performance climb together, while memorizing models saturate rapidly. CUTS is essentially an engineering response to that saturation dynamic: if the reward signal dies early, force the model to explore outputs it wouldn't otherwise consider. The 'Bounded Ratio Reinforcement Learning' paper from the same period is also relevant here — it addresses instability in the policy update step, but the problem this paper identifies sits one level upstream, in the data distribution fed to the RL loop. Neither fix alone is sufficient.

Watch whether CUTS or Mixed-CUTS holds up when applied to benchmarks with genuine headroom, like GPQA Diamond or competition-level AIME problems, where base model accuracy is lower. If the diversity gains disappear in those settings, the technique is specific to the saturation regime and not a general training improvement.