LLM critics can sabotage AutoML search without capability guardrails
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
A new audit reveals that LLM critics, increasingly deployed to guide iterative search in AutoML and prompt optimization loops, can actively degrade performance rather than improve it. Testing an 8B parameter critic on tabular datasets showed it drove search below random baseline on 7 of 8 tasks, losing up to 0.245 AUC. The failure mode depends on critic strength, dataset properties, and search budget. Controlled fault injection identified the culprit: directional errors in the critic's guidance compound across iterations, while non-directional noise remains benign under best-so-far selection. This challenges the assumption that agentic loops with LLM feedback are universally beneficial and signals practitioners need explicit capability thresholds before deploying critics in production search pipelines.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The failure mode is specific and actionable: critics don't just add noise, they systematically steer search in wrong directions, and this compounds across iterations. The distinction between benign randomness and harmful bias is the actual insight practitioners need.
This extends a pattern from the past week's coverage on LLM-guided systems. The Modelwire audit on expert disagreement (Nov 12) showed LLMs fail to selectively revise when challenged; this paper shows the same selectivity problem in automated loops, where the critic can't distinguish good guidance from bad. Similarly, the orchestration study (Nov 12) found delegation doesn't reliably improve performance. Together these suggest agentic composition is riskier than the industry narrative assumes, particularly when the guiding agent operates below a capability threshold. The difference here is that critics can actively harm, not just fail to help.
If practitioners adopt the paper's proposed capability threshold (likely tied to critic accuracy on held-out validation tasks) and report adoption rates in production AutoML systems over the next six months, that signals the finding is moving from audit to practice. If major AutoML vendors (H2O, AutoGluon, Optuna) add explicit critic validation gates to their search loops by Q2 2027, that confirms the safety concern has reached operational relevance.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Study finds LLM task delegation delivers inconsistent accuracy gains
A new study challenges the prevailing assumption that agentic LLM systems improve performance through task delegation. Researchers built datasets isolating pre- and post-cutoff knowledge to test whether delegating to external tools or other models actually boosts accuracy. The findings are sobering: delegation shows no consistent gains over single-model baselines, with effectiveness varying unpredictably by model…
MentionsLLM critics · AutoML · prompt optimization · self-refinement · 8B parameter model
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.