Benchmark reveals gaps in LLM peer review accuracy across error types
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
As LLM-based peer review moves into production use across academic venues, a new benchmark exposes critical gaps in how well these systems catch errors. Researchers injected 357 deliberate flaws spanning mathematical, numerical, and citation issues into real papers across astrophysics and ML, then tested twelve reviewer systems. The work introduces Skepthical, a multi-agent approach that grounds claims by retrieving cited sources, verifying math symbolically, and executing code. This directly challenges the reliability narrative around automated review and signals that production deployments may be catching far fewer errors than assumed, reshaping how institutions should weight LLM verdicts in editorial workflows.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
Skeptical readOur AI-generated reading of the wider context and the next developments to watch.
The key omission: Skepthical uses multi-agent grounding (source retrieval, symbolic math verification, code execution) rather than just LLM classification. But the summary doesn't specify whether this architectural difference actually outperforms the ensemble methods that already recovered 93 percent of errors in the prior benchmark, making it unclear whether we're seeing a real capability jump or just a different approach to the same problem.
This sits directly alongside the November 9 benchmark on 100 planted errors, which found that ensemble methods significantly outperform single models and that omissions remain a structural blind spot. The current work tests on a larger error set (357 vs 100) and different domains (astrophysics plus ML vs psychology), but the Anthropic auditing study from November 7 already surfaced the core tension: detection systems that catch nearly everything flood reviewers with false positives. Skepthical's grounding approach may reduce hallucination, but the summary doesn't report false-positive rates, leaving the operational feasibility question unanswered.
If Skepthical's false-positive rate stays below 5 percent while maintaining the error-catch rate above 90 percent, that would signal a genuine advance over ensemble methods. If false positives remain high despite the grounding architecture, the finding confirms that architectural sophistication alone doesn't solve the triage problem that makes automated review operationally viable at scale.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Frontier models catch 71 of 100 planted paper errors, ensemble reaches 93
Researchers benchmarked frontier LLMs and commercial AI review tools against 100 seeded errors in psychology papers, revealing a fragmented capability landscape where the best performer caught 71 errors but ensemble methods recovered 93. The finding that all seven uncaught errors were omissions rather than insertions exposes a structural blind spot in current AI review systems.…
MentionsSkepthical · arXiv · LLM peer review systems
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Auditing AI peer reviewers: a benchmark on real scientific papers”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.