Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs

The more precise claim here is that routing operates as a low-bandwidth bottleneck, and that constraint is what forces functional abstraction rather than surface-feature memorization. This reframes MoE specialization as an emergent consequence of architectural pressure, not a designed property.

This connects most directly to the generalization work covered in 'Generalization in LLM Problem Solving: The Case of the Shortest Path' (April 16). That paper found models transfer spatial patterns but collapse under recursive depth, which is consistent with a picture where routing encodes abstract relational structure while content channels stay shallow. If routing really does carry the abstract load, it would help explain why models generalize across surface variation but fail when the task demands composing those abstractions across steps. More broadly, this paper belongs to a growing body of mechanistic work trying to locate where computation actually lives in transformer-family models, a conversation that benchmark-focused coverage largely ignores.

Watch whether interpretability teams at labs running large MoE deployments (Mistral, Google with Gemini) publish follow-up ablations that selectively perturb routing signals while holding content weights fixed. If functional performance degrades faster from routing perturbation than from equivalent content perturbation, this decomposition holds up as more than a descriptive frame.