Sparse routing maps diverge from softmax in trained mixture-of-experts models
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers compared four routing mechanisms for mixture-of-experts language models, revealing that sparsity-inducing probability maps like entmax and sparsemax behave fundamentally differently from softmax during training. While these alternatives theoretically enable token-dependent expert selection through exact zeros, empirical results at 1B parameters show they diverge sharply in practice: entmax discards 30% less probability mass while rarely dropping experts, sparsemax preserves the most mass, and normmax routes 21% of tokens differently. This work matters because MoE scaling efficiency depends critically on routing behavior, and the gap between theoretical sparsity and learned routing patterns could reshape how practitioners design expert selection for larger models.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper reveals that sparsity-inducing mechanisms don't just produce fewer active experts - they fundamentally change how models learn to route tokens during training. The gap between theoretical guarantees (exact zeros) and what actually emerges (entmax rarely drops experts despite discarding less probability mass) suggests routing behavior is learned, not mechanically enforced.
This connects directly to the load-balancing work from late September. ID Balancing and CIPHER-MoE both treat routing imbalance as a systems problem requiring external correction, but this paper hints the problem may originate in the routing mechanism itself. If sparsemax and entmax produce fundamentally different learned routing patterns at 1B parameters, then load-balancing solutions designed for softmax routing may be fighting against the routing function's own learning dynamics rather than fixing a downstream artifact. The scaling laws paper from September 30 unified recurrence and sparsity efficiency, but didn't account for routing mechanism choice - this work suggests that choice cascades into scaling behavior.
If the same four routing mechanisms are tested at 10B+ parameters and the learned routing patterns remain consistent (entmax still rarely drops experts, sparsemax still preserves mass), that confirms routing behavior is stable across scale and practitioners should bake mechanism choice into architecture decisions early. If patterns shift or converge at larger scale, it signals routing behavior is scale-dependent and load-balancing solutions may need to adapt per-scale.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.LG
Control theory unlocks stable scaling for sparse mixture-of-experts models
Researchers have unified two competing load-balancing approaches for sparse Mixture-of-Experts models under a control-theory framework, revealing that DeepSeek's loss-free method and Kimi K3's Quantile Balancing operate as incomplete PID controllers. The work proposes ID Balancing, a new integral-derivative controller that addresses expert load imbalance, a critical bottleneck as parameter counts scale. This matters because extreme…
MentionsMixture-of-Experts · sparsemax · entmax · normmax · softmax
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “How Sparse Probability Maps Shape Mixture-of-Experts Routing”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.