Fidelity checks miss hallucinating agents that skip tool use entirely

The core insight isn't just that agents hallucinate; it's that they can hallucinate while passing accuracy benchmarks because those benchmarks only check outputs, not whether the agent actually performed the steps you told it to perform. This is a validation architecture problem, not a model capability problem.

This connects directly to the diagnostic-driven approach in the small-model dialogue work from late August, which identified how mechanical failures (repeated guesses, constraint violations) hide inside correct-looking outputs. Both papers argue that static correctness metrics miss execution-level breakdowns. The datasheet extraction work extends that insight to tool-using agents: you need to instrument the workflow itself, not just grade the answer. Where the dialogue paper fixed small models through targeted retraining, this one proposes observability as the fix for validation.

If teams adopting this dispatch-level instrumentation find that 10-20% of their production agents are silently hallucinating despite passing internal accuracy checks, that validates the scale of the problem. If instead the gap is under 5%, the finding matters more as a methodological refinement than a deployment risk.