Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

The deeper story here is not that Hyperball beats AdamW on a benchmark, but that it diagnoses *why* matrix-based optimizers like Muon degrade at scale: unconstrained norm growth destabilizes the very structure these methods depend on. The fix is essentially a geometric constraint, not a new learning rule.

This sits inside a quiet but accelerating wave of optimizer research that is rethinking the foundations of how large models are trained. The concurrent arXiv piece on 'Functional Gradient Descent with Adaptive Representations' (covered the same day) approaches the same underlying tension from a different angle, arguing that parameter-space optimization itself is the bottleneck and proposing function-space descent instead. Hyperball and FGD are not the same project, but they share a common premise: AdamW's dominance is a historical artifact, not a principled ceiling. Together they suggest the optimizer layer is becoming a genuine site of architectural competition, not just a tuning knob.

The real test is whether Hyperball's hyperparameter portability claim holds when applied to models significantly larger than those reported, specifically whether the same learning rate transfers cleanly from a 1B to a 10B+ scale run without manual retuning. If it does, adoption in open pretraining efforts like those using Qwen3 architectures would likely follow within two to three training cycles.