Per-example data curation framework targets LLM pretraining bottleneck
DataOrchestra introduces adaptive data curation for LLM pretraining, moving beyond fixed corpus-level rules to make per-example decisions about whether data should be dropped, preserved, or transformed. The framework orchestrates a pipeline of operations from programmatic cleaning to LLM-based rewriting, with the system generating task-specific instructions for each step. This addresses a fundamental bottleneck in model development: pretraining data quality directly shapes downstream performance, yet most pipelines apply uniform strategies across millions of examples. The work signals growing recognition that data processing deserves the same algorithmic sophistication as model architecture, potentially reshaping how labs approach the expensive pretraining phase.
Modelwire context
ExplainerThe paper doesn't just propose better data cleaning rules; it reframes data curation as a learned optimization problem where each example gets its own treatment decision. Most prior work applies the same filtering logic uniformly across billions of tokens.
This connects directly to the physics of planning paper from late July, which showed that explicit world models built through deliberate training choices outperform models trained on raw internet data. DataOrchestra operationalizes that insight by making data quality itself a design variable rather than an afterthought. Both papers treat pretraining as a system where intermediate representations matter, not just final loss curves. The medical imaging work (ClinFusion) makes a parallel argument in vision: domain-specific bottlenecks require architectural choices, not generic scaling. Here the bottleneck is data quality, and the solution is adaptive curation rather than uniform rules.
If labs report downstream task improvements that scale with curation sophistication (not just corpus size), and if those gains hold across multiple model scales and domains, the approach moves from interesting to methodologically standard. Watch whether the same teams publish follow-ups showing that DataOrchestra-curated pretraining outperforms larger raw datasets on held-out benchmarks by a measurable margin within six months.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsDataOrchestra
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.