Research·arXiv cs.CL·17h ago

RL outperforms SFT in multi-task LLM training through orthogonal parameter updates

Researchers have identified a fundamental divergence in how Supervised Fine-Tuning and Reinforcement Learning scale to multi-task LLM training. SFT induces catastrophic task interference at the parameter level, while RL maintains stable performance across diverse objectives through sparse, orthogonal gradient updates. The theoretical insight centers on gradient interference mechanics: SFT conflicts scale with absolute gradient magnitude, whereas RL bounds interference through variance constraints. This distinction has immediate implications for practitioners designing multi-objective training pipelines and suggests RL-based approaches may offer a more robust path to generalist models than traditional SFT stacking.

Modelwire context

Explainer

The paper isolates a specific mathematical reason why RL outperforms SFT in multi-task settings: RL's variance constraints naturally bound gradient conflicts, while SFT's conflicts scale linearly with gradient magnitude. This is a mechanistic insight, not just an empirical observation.

This connects directly to the GRPO bottleneck identified in the August 1st analysis of RL-group failures. That work showed RL stalls when rewards sparse out; this paper explains why RL remains stable when it does work. The multilingual ASR distillation study from August 4th also mirrors the core finding here: decoupling tasks (via language-specific teachers) then recomposing them avoids the optimization conflicts that plague joint training. Together these suggest a pattern: RL and task-specialized decomposition are becoming standard solutions to the interference problem that plagues naive multi-objective SFT.

If practitioners report measurably lower task interference when switching from SFT stacking to RL-based multi-objective training on standard benchmarks (like MMLU + GSM8K + HumanEval) within the next two quarters, the theoretical prediction holds. If interference persists despite RL adoption, the gradient mechanics proposed here may not fully explain real-world training dynamics.

Coverage we drew on

Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsLarge Language Models · Supervised Fine-Tuning · Reinforcement Learning

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.