Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers have built the first unified benchmark for detecting hallucinations at the span level across code, tool outputs, and structured documents, moving beyond the natural-language-only focus of prior RAG evaluation work. A fine-tuned Qwen 3.5-2B model achieves 0.689 span-F1 on the combined test set and substantially outperforms existing baselines on code-agent tasks. This matters because production AI systems increasingly ground reasoning in heterogeneous sources like repositories and CLI output, yet hallucination detection methods remain calibrated for prose. The benchmark and detector provide a foundation for building more reliable code-aware retrieval systems.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The key gap this fills is not just accuracy but scope: prior hallucination detection benchmarks treat grounding as a prose-to-prose comparison problem, but production code agents routinely reason over CLI output, stack traces, and repository diffs where token-level fidelity means something structurally different than in natural language.
This connects directly to the FinKG-News work covered the same day, which found that automated hallucination detection remains unreliable even when inputs are structured and grounded. That paper flagged the problem in financial documents; this one attacks the same failure mode in code and tool output. Together they sketch a pattern: as AI systems move into higher-stakes, heterogeneous-source environments, the hallucination detection tooling built for prose RAG simply does not transfer. The MAGNET story from the same period is also relevant, where the ATLAS verifier handles scene-level narrative consistency, reinforcing that domain-specific detection architectures are becoming a recurring design choice rather than an edge case.
Watch whether the benchmark gets adopted by code-agent evaluation suites like SWE-bench or similar within the next two quarters. If it does, the Qwen 3.5-2B detector results will face real stress-testing against diverse agent trajectories and the 0.689 span-F1 figure will either hold or reveal the limits of the training distribution.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Evidence-Supported Credit Risk Report Generation Using News-Centric Financial Knowledge Graphs
Researchers have developed FinKG-News, a framework that grounds LLM-based financial analysis in structured knowledge graphs extracted from news events. The system links real-world occurrences to company data, then uses this evidence layer to generate credit risk reports across multiple financial dimensions. A critical finding: automated hallucination detection remains unreliable even with grounded inputs, underscoring that…
MentionsQwen 3.5-2B · arXiv
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.