Effective learning rate ratio predicts loss dynamics across model scales
Researchers have identified a fundamental scaling law in language model training: loss trajectories converge when the ratio of learning rate to parameter norm (effective learning rate) is held constant, regardless of the individual values of either variable. This finding holds across optimizers, architectures, and model scales with collapse errors typically below natural seed variation. The work reveals that weight decay and optimization geometry primarily influence training dynamics through their effect on this ratio, suggesting practitioners can predict and control training behavior more precisely by focusing on this single composite metric rather than tuning learning rate and model size independently.
Modelwire context
ExplainerThe paper's real contribution is showing that effective learning rate acts as a sufficient statistic for loss dynamics across otherwise independent variables. This means the collapse works not just within a single setup but across optimizers and architectures, suggesting the ratio is closer to a fundamental invariant than a convenient coincidence.
This connects directly to the broader pattern in recent work around diagnostic opacity and measurement reliability. Just as the FID paper from late August exposed how dominant metrics hide distributional failures, this research reveals that practitioners have been tuning two variables when one composite metric governs the outcome. Both papers address a similar problem: existing tools (metrics, hyperparameter choices) obscure the actual mechanism driving behavior. The difference is scope: FID hides what generative models produce, while this work clarifies what controls how language models learn.
If major training frameworks (PyTorch, JAX) add effective learning rate as a first-class logged metric within six months, adoption will signal the community accepts this as actionable. If practitioners report reproducible loss curves using only this ratio across different model sizes within a year, the finding has moved from theoretical to operational.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsLanguage models · Effective learning rate · Weight decay · Hyperball
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.