Sparse models degrade faster on repeated training data than dense networks

The paper isolates a specific failure mode of sparsity itself, not just data exhaustion. MoE models don't just struggle with less data; they actively memorize repeated examples at rates that scale with total model size, not active parameters. This suggests the efficiency math labs have been using to justify sparse scaling may not hold under realistic data recycling.

This connects directly to the distribution shift framework from the General Quantification paper (early September). That work provided tools to measure and diagnose domain mismatch in production systems. This MoE finding identifies a specific architectural vulnerability to one type of shift: repeated data is a form of covariate collapse. Labs now have both the diagnostic capability and evidence that their chosen architecture may amplify the problem. The implication is that sparse model deployments may need tighter monitoring for overfitting signals than dense alternatives, or require active data augmentation strategies that dense models tolerate better.

If Anthropic, DeepSeek, or Meta publish ablations showing MoE models trained on deduplicated corpora (vs. recycled data) recover parity with dense models on downstream tasks within the next six months, the finding is architecturally significant. If they don't publish, or if sparse models continue to underperform despite deduplication efforts, it signals the industry is absorbing this cost silently rather than redesigning training pipelines.