LLM judges fail to validate correct tool arguments without explicit text references
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers have identified a systematic failure mode in LLM-based tool-call evaluation: judges consistently underrate correct argument values when those values lack explicit textual grounding in their input. The gap emerges across seven different judge models, including advanced systems, and persists even when truncation is controlled for. The finding reveals that evaluators rely heavily on surface-level reference matching rather than semantic reasoning, creating a blind spot for inferred or schema-derived parameters. This has direct implications for agent reliability and suggests current evaluation frameworks may systematically penalize correct but non-obvious tool invocations.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The derivation gap isolates a particular mechanism driving judge failure: models penalize correct answers that require inference or schema-based reasoning rather than explicit textual copying. This is narrower and more actionable than general judge unreliability, pointing to a fixable architectural preference rather than random brittleness.
This builds directly on the evaluation brittleness audit from early October ("All Verdicts are Not Equal") and the information design study ("The Anatomy of a Blind Spot"), which showed that what judges receive shapes their verdicts. The derivation gap reveals a specific failure mode: judges don't just struggle with missing context, they actively downweight correct reasoning that isn't explicitly grounded in input text. This connects to the TRACE finding that superficial changes (like tool renaming) shift scores, suggesting judges rely on surface matching rather than semantic validation. The pattern across seven models indicates this isn't random noise but a systematic preference baked into how LLMs process evaluation tasks.
If the same seven judge models show improved performance on a held-out test set where correct arguments are explicitly paraphrased into the input context (versus requiring inference), that confirms the derivation gap is the bottleneck rather than a symptom of deeper reasoning failure. If performance doesn't improve, the problem lies upstream in how judges approach semantic validation itself.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
LLM judges fail reliability audit, undermining evaluation standard
A systematic audit of LLM-as-a-Judge evaluation exposes fundamental brittleness in a practice now treated as industry standard. Testing six frontier models across multiple conditions reveals that verdicts shift even at zero temperature, position bias flips majority outcomes on hard tasks, and the most consistent judge achieves reliability by repeating errors while matching ground truth only…
MentionsCLM · Jev
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.