Research·arXiv cs.CL·1d ago

Sparse MoE hyperparameter laws diverge from dense model scaling

Researchers have identified fundamental gaps in how hyperparameter scaling laws apply to sparse Mixture-of-Experts models, a critical finding for practitioners scaling MoE architectures. The study, grounded in 1,800 pre-training experiments consuming 200,000 H100-equivalent GPU-hours, reveals that learning rate and batch size optimization cannot be reliably transferred across sparsity levels using conventional parameter-count metrics. This work resolves conflicting prior findings by mapping distinct scaling regimes, directly impacting how teams will tune ultra-sparse models in production. For infrastructure and model teams, the implication is clear: existing hyperparameter recipes break down at high sparsity, requiring new empirical protocols or model-specific tuning strategies.

Modelwire context

Explainer

The critical omission: this work doesn't just say hyperparameters don't transfer across sparsity levels, it maps the distinct regimes where they break. That distinction matters because it suggests practitioners can't just apply a blanket 'retune everything' rule; instead, they need to identify which regime their target sparsity falls into.

This connects directly to the Chimaera paper from today on mixture-of-graph-experts, which also uses expert routing to handle structural diversity across datasets. Both papers grapple with the same underlying problem: when you introduce routing or sparsity, the optimization landscape fragments. The difference is Chimaera addresses it architecturally (flexible expert composition), while this work addresses it empirically (new tuning protocols). Together, they suggest that sparse and routed systems require fundamentally different scaling assumptions than dense baselines, a theme absent from earlier transformer theory work like the C-RASP length generalization paper.

If major labs (Anthropic, DeepSeek, or xAI) publish MoE training recipes in the next 6 months that explicitly cite sparsity-dependent hyperparameter regimes rather than applying dense-model conventions, this finding has moved from research into practice. If they don't, it signals the empirical cost of retuning hasn't yet justified the infrastructure investment.

Coverage we drew on

Chimaera: A Mixture-of-Graph-Experts Architecture for Cross-Task and Cross-Dataset Graph Learning · arXiv cs.LG

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsMixture-of-Experts · MoE

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Hyperparameter Scaling Laws Across MoE Sparsity”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.