Slicing and Dicing: Configuring Optimal Mixtures of Experts

The practical implication buried in the summary is that MoE configuration has largely been a craft skill, with labs making educated guesses about expert count and routing in isolation. This study is the first to treat those choices as a joint experimental space, which means prior intuitions about, say, increasing expert count while holding dimensions fixed may have been systematically misleading.

This connects to the on-policy distillation work covered the same day ('Learning to Foresee'), which found that training efficiency gains emerge from gradient concentration in dominant subspaces rather than explicit architectural guidance. Both papers are pushing toward the same underlying question: how much of model performance is determined by architectural choices made before training begins, versus dynamics that emerge during the run itself. The MoE factorial study answers that question at the pretraining configuration level, while the distillation paper answers it at the optimization level. Together they suggest practitioners have more leverage over final model quality through upfront design than recent scaling orthodoxy has assumed.

Watch whether a major lab (DeepSeek or Mistral are the obvious candidates given their public MoE work) publishes an architecture update within six months that explicitly cites total parameter scaling as a design constraint, which would confirm this empirical baseline is being adopted in production rather than remaining a research artifact.