From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models

The finding isn't just that current pruning heuristics are imprecise, it's that the metrics practitioners rely on carry zero predictive signal for actual expert importance, meaning the entire measurement vocabulary for MoE optimization may need to be rebuilt from scratch rather than refined.

This connects directly to the 'Continual LLM Upcycling' paper from the same day, which proposes converting dense models to sparse ones through learned routing trained alongside standard objectives. That work assumes routing behavior during training is a meaningful signal for which activations matter. The causal audit covered here throws that assumption into question: if routing statistics don't predict importance under intervention, then a sparsity recipe built on routing logic may be optimizing a proxy that doesn't track the thing it claims to track. More broadly, the 'When the Chain of Thought Knows Better' paper from the same batch surfaces a parallel problem in a different domain, where observable outputs systematically misrepresent internal states. Both papers are pointing at the same structural issue: the field has built evaluation and optimization pipelines on observational proxies that don't survive causal scrutiny.

Watch whether the authors or independent groups apply this intervention framework to the routing assumptions inside the Upcycling paper's predictor-gated architecture. If importance rankings diverge from routing weights there too, that would suggest the correlation-causation gap is a property of sparse routing mechanisms generally, not just post-hoc pruning contexts.