Muon as a Residual Connection

The contribution here is not a new optimizer but a new explanation for why an existing one works, which is arguably more useful: a mechanistic account lets practitioners reason about when Muon's trade-offs are worth accepting rather than treating it as a black-box performance win.

The tension this paper names, trading local accuracy for global coherence, echoes a pattern visible across recent coverage. The 'Diffeomorphic Optimization' paper from the same day addresses a structurally similar problem: standard gradient descent ignores the geometry of the space it operates in, and correcting for that geometry changes what 'progress' means at each step. Both papers are essentially arguing that the coordinate system you optimize in is not neutral. The 'GSRQ' work on KV cache quantization also surfaces a related geometric concern, centroid shrinkage in high dimensions, where naive local decisions degrade global representational fidelity. The common thread is that ML systems at scale increasingly require optimizers and compression schemes that account for structure across the full computation graph, not just at the point of update.

The practical test is whether practitioners training large models can identify specific layer types or depth ranges where Muon's orthogonalization penalty is clearly worth paying, producing a concrete configuration heuristic. If no such heuristic emerges from follow-up ablations within the next few months, the mechanistic framing remains theoretically tidy but operationally inert.