Per-example data curation framework targets LLM pretraining bottleneck
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
DataOrchestra introduces adaptive data curation for LLM pretraining, moving beyond fixed corpus-level rules to make per-example decisions about whether data should be dropped, preserved, or transformed. The framework orchestrates a pipeline of operations from programmatic cleaning to LLM-based rewriting, with the system generating task-specific instructions for each step. This addresses a fundamental bottleneck in model development: pretraining data quality directly shapes downstream performance, yet most pipelines apply uniform strategies across millions of examples. The work signals growing recognition that data processing deserves the same algorithmic sophistication as model architecture, potentially reshaping how labs approach the expensive pretraining phase.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper doesn't just propose better data cleaning rules; it reframes data curation as a learned optimization problem where each example gets its own treatment decision. Most prior work applies the same filtering logic uniformly across billions of tokens.
This connects directly to the physics of planning paper from late July, which showed that explicit world models built through deliberate training choices outperform models trained on raw internet data. DataOrchestra operationalizes that insight by making data quality itself a design variable rather than an afterthought. Both papers treat pretraining as a system where intermediate representations matter, not just final loss curves. The medical imaging work (ClinFusion) makes a parallel argument in vision: domain-specific bottlenecks require architectural choices, not generic scaling. Here the bottleneck is data quality, and the solution is adaptive curation rather than uniform rules.
If labs report downstream task improvements that scale with curation sophistication (not just corpus size), and if those gains hold across multiple model scales and domains, the approach moves from interesting to methodologically standard. Watch whether the same teams publish follow-ups showing that DataOrchestra-curated pretraining outperforms larger raw datasets on held-out benchmarks by a measurable margin within six months.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Controlled experiments reveal world models beat raw data for agent planning
Researchers have constructed a controlled experimental framework to isolate how foundation model agents develop long-horizon planning capabilities across pre-training and post-training phases. The work reveals that explicit world models built through chain-of-thought state transitions outperform models trained on raw internet data, and that atomic skill composition alone fails to generalize across multi-step tasks. This challenges…
MentionsDataOrchestra
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.