AutoData uses agents to automate pre-training data selection

The key insight is that AutoData treats data curation as a searchable algorithm space rather than a fixed weighting scheme. Prior work assumed domain experts should manually decide which data sources matter; AutoData lets an agent discover which combinations of scoring, stratification, and sampling rules work best for a given validation set.

This connects directly to the reward modeling and curriculum learning work from this week. F2DR and MATCH both tackle the problem of learning what to optimize for across multi-step pipelines, but they focus on inference-time behavior (search workflows, tool use). AutoData pushes that same principle backward into the pre-training layer itself. The Dictionary-Constrained Grapheme-to-Phoneme paper also used LLM-generated data with constraints, but that was manual constraint design; AutoData automates the discovery of which constraints and sampling rules matter. Together, these papers suggest a broader trend: the ML stack is moving from fixed heuristics (manual domain weighting, fixed curriculum schedules, hand-coded constraints) toward learned, adaptive policies at every layer.

If AutoData's discovered algorithms transfer across model scales (e.g., the optimal data selection policy for a 7B model also improves a 70B model without retraining), that confirms the approach captures genuine data properties rather than overfitting to a specific validation set. If not, the method is a form of expensive per-scale tuning with limited practical value.