Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works
Researchers identified a critical failure mode in Group Relative Policy Optimization (GRPO), a core algorithm for training language models on verifiable tasks like math. When all responses in a batch are correct or all wrong, the advantage signal collapses to zero, starving the policy of learning gradients. The team proved this degeneracy rate exceeds theoretical predictions and observed it empirically in Qwen3.5 training. They propose a minimal fix using a fixed-reference sign-based advantage that maintains learning signal by optimizing for at least one correct sample per group. This addresses a fundamental instability in a widely-used RL training method, with direct implications for math reasoning and code generation workloads.
Modelwire context
ExplainerThe paper's most underreported finding is quantitative: the degeneracy rate in real training runs exceeds what theory predicts, meaning practitioners using GRPO on math or code tasks may have been losing more training signal than any back-of-envelope estimate would suggest. The fix is deliberately minimal, which is itself a signal that the authors prioritized adoption over novelty.
This connects to a pattern visible in recent Modelwire coverage: training and inference pipelines for language models are accumulating silent failure modes that only surface under careful empirical scrutiny. The grammar-constrained decoding paper from the same day ('Future Validity is the Missing Statistic') documented a case where deployed systems sample from the wrong distribution without any visible error signal. The GRPO gradient starvation problem is structurally similar: the training loop continues, losses look reasonable, and the failure is invisible until someone measures the advantage signal directly. Both papers argue that the gap between theoretical guarantees and empirical behavior is larger than the field has assumed.
Watch whether the Qwen or DeepSeek teams publish ablations comparing the sign-based advantage fix against standard GRPO on MATH-500 or AIME within the next two months. If the fix holds at scale on those benchmarks, expect rapid adoption in open post-training recipes.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsGRPO · Qwen3.5-9B · GSM8K · Group Relative Policy Optimization
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.