UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models

The core problem UDM-GRPO addresses is a distribution mismatch: standard GRPO was designed for autoregressive models, so plugging it directly into a diffusion process produces training instability because the rollout trajectories don't resemble what the model saw during pretraining. The reconstruction step is the actual contribution, not the RL application itself.

None of the related stories on Modelwire connect cleanly to this work. The closest thematic neighbor is the benchmarking piece on optimizers for tabular deep learning, which also grapples with whether a training algorithm designed for one setting transfers reliably to another. But UDM-GRPO sits in a fairly distinct corner: applying policy optimization to generative models with discrete token spaces and non-autoregressive structure. The broader conversation about RL fine-tuning for language models is active elsewhere, but our recent archive doesn't reflect it directly.

Watch whether the trajectory reconstruction approach holds when applied to larger discrete diffusion models trained on code or protein sequences. If benchmark gains persist at that scale, the distribution-alignment framing becomes a general recipe worth adopting; if they degrade, the method may be sensitive to the specific noise schedule used in pretraining.