Gauge symmetry explains why Adam and gradient descent find different solutions
A new theoretical framework reveals why gradient descent and Adam optimizer diverge on factored matrix models despite identical initialization. The culprit is gauge symmetry: gradient descent exploits the loss function's invariance under certain transformations to converge toward low-rank solutions, while Adam's coordinate-wise updates break this symmetry and fail to inherit the same implicit bias. The work characterizes which optimizers preserve gauge equivariance (gradient descent, momentum, Muon, Shampoo) versus those that don't (Adam, RMSProp). This distinction matters for practitioners tuning optimization on large-scale factored models and informs the design of future optimizers that could recover gradient flow's efficiency properties.
Modelwire context
ExplainerThe paper's core insight is that optimizer choice encodes geometric assumptions about the loss landscape. Adam doesn't fail because it's broken; it fails because it's solving a different optimization problem than gradient descent, one that doesn't preserve the symmetries the loss function exploits.
This connects to the inference efficiency push from the Opt.Gear report (August 2) and Baseten's work on deployment optimization (August 3), but from the opposite end. Those pieces focus on making trained models faster; this paper shows that which optimizer you pick during training fundamentally shapes what solution you converge to. The characterization of gauge-equivariant optimizers (gradient descent, Muon, Shampoo) versus non-equivariant ones (Adam, RMSProp) matters because it means practitioners can't just swap optimizers for speed without changing the model's implicit inductive bias. It's a reminder that training efficiency and inference efficiency are decoupled problems.
If practitioners report that Muon or Shampoo recover Adam-like wall-clock training time while preserving gradient descent's low-rank implicit bias on production factored models over the next 6 months, this moves from theory to practice. If they don't, the work remains a useful diagnostic tool but doesn't shift how teams actually tune.
Coverage we drew on
- Opt.Gear Technical Report · arXiv cs.CL
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsAdam · Gradient Descent · RMSProp · Muon · Shampoo
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “The Loss Does Not See the Basis, but Adam Does”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.