Repeated tokens lose half their value after critical epoch threshold
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
A new framework quantifies the efficiency loss when language models train on repeated data, establishing that a second epoch on the same corpus recovers nearly as much value as fresh data, but diminishing returns accelerate sharply beyond a critical threshold tied to training budget per parameter. This finding reshapes pretraining economics: as data scarcity pressures labs to recycle corpora, the research provides a principled way to trade off repetition against model scaling, suggesting that epoch count matters far more than model size in determining when reuse becomes wasteful. For practitioners balancing finite unique data against compute budgets, this offers a quantitative anchor for epoch scheduling decisions.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
Analyst takeOur AI-generated reading of the wider context and the next developments to watch.
The paper doesn't just measure repetition cost; it establishes a critical threshold beyond which epoch count becomes the dominant constraint on efficiency, not model size. This inverts conventional scaling wisdom and suggests labs have been over-investing in parameter count relative to data reuse strategy.
This directly complements the looped MoE scaling laws from late September, which unified recurrence and sparsity into a single efficiency framework. Where that work showed how to trade parameters and active compute, this paper adds a third lever: data repetition scheduling. Together they sketch a more complete efficiency frontier. The finding also echoes the precision-count trade-off logic from the BreadthKV work on KV compression (early October), where the constraint shifts from raw capacity to strategic allocation within a fixed budget. Labs now have quantitative anchors for three separate scheduling decisions: expert routing, token retention, and epoch count.
If a major lab (Anthropic, DeepSeek, or Meta) publishes pretraining details for a model trained after this paper's October release and reports epoch counts significantly higher than prior work while holding model size flat, that confirms the framework is reshaping practice. Conversely, if published training runs continue to scale parameters faster than epochs, the paper's prescriptions haven't moved incentives yet.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Unified scaling laws bridge looped transformers and sparse experts
Researchers have unified scaling laws for two previously separate efficiency strategies in large language models: recurrent depth and sparse expert routing. The work models how looped transformers and mixture-of-experts interact during scaling, showing that sparsity amplifies the computational gains from recurrence. This matters because it gives practitioners a unified framework for trading off model parameters,…
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.