Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization

The key insight is that heterogeneous reward signals don't just need scaling; they need their correlation structure removed before aggregation. RDPO uses Mahalanobis whitening to decorrelate rewards, not just normalize them. This is distinct from standard multi-task learning approaches that treat reward heterogeneity as a magnitude problem alone.

This connects directly to the distillation failure mode identified in 'Prefix Teach, Suffix Fade' from earlier this week. Both papers surface how naive aggregation of multiple learning signals (whether from teacher feedback or heterogeneous rewards) can paradoxically harm training. Where that work showed dense supervision across full sequences degrades performance, RDPO shows that mixing uncorrelated reward types destabilizes advantage estimation. The underlying problem is similar: signal quality matters more than signal quantity, and naive combination breaks optimization. Both suggest practitioners need to be selective about which signals to combine and how.

If RDPO maintains performance gains when applied to instruction-following models with genuinely conflicting objectives (e.g., helpfulness vs. safety vs. length constraints), rather than just the LongCat-Flash benchmark, that confirms the decorrelation principle generalizes. If follow-up work shows perplexity-matched baselines using standard reward aggregation fail on the same tasks, that would validate the claim that correlation noise is the bottleneck, not just scale.