LLMs fail to validate evidence in security audits, study finds
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
Researchers measured whether LLMs can distinguish genuine from fabricated evidence when auditing code vulnerabilities. Testing 916 function pairs across 11 model configurations from four vendors, they found that true static-analysis output barely improved accuracy (+0.02), while false evidence sometimes degraded performance asymmetrically depending on what the false claim asserted. This reveals a critical gap in how production LLM systems handle external tool outputs, suggesting that combining models with analyzers may create a false sense of rigor without corresponding gains in reliability or skepticism.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
Skeptical readOur AI-generated reading of the wider context and the next developments to watch.
The asymmetry is the buried detail. True static-analysis output barely moved the needle (+0.02 accuracy), but false evidence didn't hurt uniformly. Depending on what the false claim asserted, models sometimes performed worse. This suggests LLMs aren't just ignoring external tools; they're selectively trusting them in ways that correlate with their own priors rather than evidence quality.
This connects directly to 'Prompt Sensitivity in Tool-Augmented Model Comparisons' from last week, which found that tool integration amplifies rather than stabilizes model behavior. Here we see the mechanism: when a tool output aligns with a model's existing bias, the model defers; when it contradicts, the model either ignores it or trusts it less. The pattern also echoes 'Speak vs. Know' from October 20th, where models' stated reasoning diverged from their actual distributions. In both cases, what the model claims to do (use evidence, apply logic) doesn't match what it actually does (pattern-match to priors).
If the researchers test whether models show higher deference to false evidence that confirms common vulnerability patterns versus false evidence that contradicts them, that would confirm whether the asymmetry is driven by prior bias rather than evidence evaluation. Watch for a follow-up testing the same 916 pairs with evidence that contradicts versus aligns with typical C/C++ vulnerability distributions.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.LG
Tool use widens prompt sensitivity gaps in model rankings
A new evaluation protocol reveals that tool use in language models amplifies rather than stabilizes ranking volatility across question phrasings. Researchers tested six open models across five domains and found that while tools improve absolute accuracy when programs can answer directly, they widen the sensitivity gap between model pairs to different prompt wordings. The effect…
MentionsPrimeVul · C/C++
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Trusting the Tool: Asymmetric Deference of LLM Vulnerability Auditors to Static-Analysis Evidence”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.