Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts

The core insight BAR exploits is that most post-training failures aren't optimization failures — they're interference failures, where updating one capability quietly degrades another. By routing domain updates through separate expert modules rather than shared weights, the method sidesteps that interference entirely, which is a different framing than simply 'cheaper training.'

This connects directly to a cluster of post-training research appearing on Modelwire this week. 'Too Correct to Learn' identified how reinforcement learning on saturated data collapses into homogeneous outputs, and 'Bounded Ratio Reinforcement Learning' proposed tighter theoretical constraints on policy updates. Both papers are wrestling with the same underlying tension: continued training on capable models tends to break things in subtle ways. BAR approaches that tension from the architecture side rather than the objective side, which makes it complementary rather than redundant. Together, these papers suggest the field is converging on a shared diagnosis — that naive continued training is fragile — while still disagreeing sharply on the right fix.

The real test is whether BAR's linear-cost scaling holds when the number of domains grows past the handful typically used in MoE benchmarks. If a follow-up demonstrates stable performance across 20 or more distinct domains without router collapse, the architectural claim becomes substantially stronger.