Systematic decomposition reveals no universal automated discovery harness

The real buried lede is methodological: the researchers ran over 3.1 million LLM rollouts specifically to separate genuine performance signal from variance, which means prior comparisons between frameworks like OpenEvolve and TTT-Discover may have been drawing conclusions from noise rather than design. The scale of the repeated-trial analysis is itself an implicit critique of how the field has been benchmarking these systems.

This is largely disconnected from recent activity in our archive, as Modelwire has no prior coverage of automated discovery frameworks or LLM-driven search harnesses to anchor against. The work sits within a broader conversation in the ML research community about evaluation rigor, one that has surfaced repeatedly around benchmark inflation and composite system comparisons, but we have not yet tracked that thread directly. Readers following AI coding or scientific discovery tools will recognize the practical stakes: if no off-the-shelf pipeline generalizes, every deployment is effectively a custom tuning problem.

Watch whether OpenEvolve or TTT-Discover maintainers respond with ablation studies of their own that either replicate or contest these findings on additional problem domains within the next six months. If neither team engages, that silence will itself say something about how seriously the automated discovery field takes evaluation discipline.