Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching

The key contribution is methodological: the researchers isolate the architectural advantage of MoE by running three separate experiments (sparse MoE, dense matched on active params, dense matched on total params) rather than conflating routing overhead with genuine sparsity gains. Prior comparisons often mixed these variables, making it unclear whether MoE wins because of routing efficiency or simply because practitioners were comparing unequal parameter budgets.

This connects directly to the MinT infrastructure paper from earlier today, which manages both dense and MoE variants at scale. MinT's system assumes MoE and dense models have different deployment trade-offs, but this experiment clarifies what those trade-offs actually are under controlled conditions. The stateful inference work on the same date addresses a separate efficiency layer (prefill latency), orthogonal to the architecture choice this paper examines. Together they suggest the 2026 efficiency frontier involves stacking multiple optimizations: architecture selection (this paper), inference patterns (stateful transformers), and serving infrastructure (MinT).

If researchers replicate this comparison at 1B+ parameters and the MoE advantage persists, the active-parameter-matched result becomes a strong signal for production adoption. If the gap narrows or reverses at larger scales, it suggests MoE's benefits are primarily a small-scale artifact. Watch whether the Mixtral and Switch teams cite this work in their next technical reports as validation or pushback.