Skip to content
Modelwire
Subscribe

GPT and Claude overweight appendix content in paper reviews

Source published ·Modelwire updated

Original coverage: arXiv cs.CL ↗·How Modelwire adds context

The development

GPT and Claude models exhibit a systematic blind spot when reviewing academic papers, fixating on appendix content for roughly half their critiques while human reviewers draw only 8.4% of feedback from supplementary material. This divergence reveals a fundamental misalignment in how LLMs weight information hierarchy compared to domain experts, with implications for peer review integrity as AI-assisted evaluation becomes standard practice. The finding suggests current models lack the contextual reasoning to distinguish between primary contributions and supporting details, a gap that could distort publication decisions if unchecked.

Modelwire’s AI-generated summary of coverage from arXiv cs.CL.

Modelwire analysis

Explainer

Our AI-generated reading of the wider context and the next developments to watch.

The appendix bias reveals a structural problem distinct from error-detection gaps: LLMs don't just miss flaws, they actively misweight what counts as evidence of contribution. This isn't about catching planted errors or soundness assessment, but about how models parse the rhetorical architecture of academic papers.

Prior coverage has documented that AI reviewers struggle with omissions (the November 9 benchmark study) and that production systems catch fewer errors than assumed (the Skepthical audit from November 12). This story adds a layer: even when LLMs engage with a paper, they're reading it through a distorted lens. The PaperCritic framework from today sidesteps subjective judgment by targeting soundness, but this appendix bias suggests that even objective criteria may be compromised if the model can't distinguish signal from noise in the paper's structure. The problem predates training data and affects how any AI reviewer would weight evidence.

If ACL or EMNLP (both mentioned as venues) publish guidelines on AI-assisted review workflows in the next two quarters, check whether they explicitly flag appendix-heavy feedback as a red flag or require human re-weighting of LLM critiques. If they don't, it signals the field hasn't internalized this finding as actionable.

This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error

Coverage behind this analysis

These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.

  1. ·arXiv cs.CL

    Benchmark reveals gaps in LLM peer review accuracy across error types

    As LLM-based peer review moves into production use across academic venues, a new benchmark exposes critical gaps in how well these systems catch errors. Researchers injected 357 deliberate flaws spanning mathematical, numerical, and citation issues into real papers across astrophysics and ML, then tested twelve reviewer systems. The work introduces Skepthical, a multi-agent approach that…

    Read Modelwire coverage →Original source ↗

MentionsGPT · Claude · ACL · EMNLP · COLM · ICLR

MW

How this coverage is produced

Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “AI Reviews Are Biased Towards the Appendix”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.

More to explore

Benchmark reveals gaps in LLM peer review accuracy across error types

arXiv cs.CL·

Frontier models catch 71 of 100 planted paper errors, ensemble reaches 93

LLM-generated medical notes corrupt downstream AI decision-making

GPT and Claude overweight appendix content in paper reviews · Modelwire