New benchmark exposes where LLM agents fail at tool use

The 86 percent ceiling is less interesting than what sits beneath it: the benchmark distinguishes between a model that calls the wrong tool and one that calls the right tool but ignores its output entirely, two failure modes that look identical in aggregate scoring but require completely different fixes in production.

This connects directly to the span-level hallucination detection work covered here in early July ('Beyond Document Grounding'), which similarly argued that existing evaluation methods collapse meaningfully different error types into a single score. Both papers are pushing toward the same structural critique: that aggregate metrics give practitioners false confidence about where their systems actually break. The forgetting audit work ('Auditing Forgetting in Limited Memory Language Models') made the same point in a different register, showing that post-deletion metrics mask persistent knowledge pathways. A pattern is forming across recent arXiv coverage where the benchmark itself is the contribution, not a model improvement, signaling that the field is in a diagnostic phase before it can make reliable progress on reliability.

Watch whether any of the 19 evaluated model providers respond by publishing targeted improvements on the specific failure categories ToolFailBench isolates. If fine-tuning runs appear that address fabricated outputs without degrading tool-call rates, that confirms the taxonomy is actionable rather than just descriptive.