AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments

The key insight AdaMeZO contributes is not just memory savings but a demonstration that adaptive learning rates can be approximated without storing the first and second moment vectors that normally make Adam so expensive, which sidesteps what has been the assumed fundamental cost of adaptive optimization at scale.

This sits inside a broader cluster of work on making large model training and adaptation cheaper at the hardware level. The 'Randomized Subspace Nesterov Accelerated Gradient' paper covered here on the same day attacks a related problem from the gradient computation side, targeting forward-mode differentiation and distributed training bottlenecks. Both papers are essentially chipping away at the same constraint: that modern optimizers were designed assuming abundant GPU memory and clean gradient access. AdaMeZO approaches this from the fine-tuning angle, where practitioners are most immediately budget-constrained, while the subspace Nesterov work targets pretraining-scale infrastructure. Together they suggest the optimization research community is converging on a shared goal of decoupling algorithmic quality from memory footprint.

Watch whether AdaMeZO's convergence gains replicate on instruction-tuning benchmarks beyond the paper's reported tasks. If independent groups reproduce Adam-comparable results on FLAN or similar suites within the next two quarters, the method is likely to get absorbed into mainstream fine-tuning libraries like Hugging Face PEFT.