Research·arXiv cs.CL·11h ago

Paraphrasing beats repetition in LLM pre-training efficiency

Researchers isolate how LLMs absorb factual knowledge during training, finding that diverse reformulations of the same concept outperform simple repetition when token budgets are constrained. The work challenges conventional wisdom about pre-training efficiency: paraphrasing and auxiliary views prove more effective than raw duplication, even for memorization tasks, and this benefit holds regardless of the source model's quality. These findings reshape thinking about data curation and training efficiency, suggesting practitioners should prioritize conceptual variety over volume to maximize learning within fixed computational budgets.

Modelwire context

Explainer

The paper isolates a specific mechanism: under token constraints, conceptual diversity during pre-training outperforms simple data duplication for factual absorption. Critically, this benefit persists even when the base model quality varies, suggesting the effect is robust rather than model-dependent.

This connects directly to the data curation thread from early September. The difficulty-aware curation work on document VLMs (Sept 1) showed that targeted data selection beats raw scale for specialized tasks. This new finding generalizes that principle to the pre-training phase itself: the composition of training data matters more than volume when budgets are fixed. It also echoes the enterprise consolidation study (Sept 1), which used production telemetry to guide post-training allocation. Here, the implication is that pre-training data should be curated with similar intentionality, not treated as a static corpus to be repeated.

If practitioners implementing this finding report measurable perplexity gains on held-out factual benchmarks within the next two quarters while holding token count constant, that validates the mechanism. Conversely, if downstream fine-tuning performance doesn't improve proportionally to pre-training gains, it suggests the benefit is narrow to memorization and doesn't transfer to reasoning tasks.

Coverage we drew on

Closing Cost-Quality Gap in Document VLMs: Difficulty-Aware Data Curation and Quality-Adjusted Deployment Economics · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsLarge language models

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.