Beyond Distribution Sharpening: The Importance of Task Rewards

The paper's most consequential contribution may be the framing question it forces on the field: when a model improves after RL training, is it learning something genuinely new or just becoming more likely to produce what it already knew? That distinction matters enormously for how labs should allocate compute between pretraining and post-training stages.

This connects directly to the IG-Search paper from April 16, which proposed step-level information gain rewards as a way to avoid gradient collapse in search-augmented reasoning. That work assumed task rewards were the right signal but didn't interrogate whether the gains reflected new capability or surfaced latent knowledge. The current paper provides a theoretical basis for answering that question, and the answer has implications for every reward-design paper in the recent queue, including the drug discovery RL benchmark covered the same day. If distribution sharpening reliably hits unfavorable optima, then papers that rely on it as a baseline comparison may be understating the gap between approaches.

Watch whether labs publishing RLHF or RLAIF ablations in the next two quarters begin reporting distribution sharpening as a distinct baseline condition. If that separation becomes standard practice in major venue submissions, it signals the field has accepted this paper's framing as settled.