Skip to content
Modelwire
Subscribe

Model version swaps flip ICLR acceptance verdicts by 79 points

Source published ·Modelwire updated

Original coverage: arXiv cs.CL ↗·How Modelwire adds context

The development

Frontier LLM providers are silently rolling out model updates that dramatically shift evaluation outcomes, undermining the reliability of peer review at major venues. Researchers auditing Claude Opus and Gemini versions across ICLR 2027 submissions found acceptance rates swinging by 43 to 79 percentage points between model iterations, with no papers improving under newer versions. The instability persists even when controlling for review text, suggesting fundamental scoring drift. This exposes a structural vulnerability in AI-driven academic gatekeeping: benchmark scores treated as stable measurements are actually hostage to provider release cycles, creating invisible bias in which research gets published.

Modelwire’s AI-generated summary of coverage from arXiv cs.CL.

Modelwire analysis

Analyst take

Our AI-generated reading of the wider context and the next developments to watch.

The critical detail buried in the summary: acceptance rates swing wildly not because reviewers change their minds, but because the scoring model itself drifts between versions. No papers improve under newer versions, suggesting this isn't random noise but directional bias baked into provider release cycles.

This connects directly to the audit framework from mid-November showing LLM peer reviewers fail to selectively revise decisions when challenged (the expert disagreement study). That work exposed a gap between apparent responsiveness and actual reasoning. This new finding suggests the gap runs deeper: the reviewers aren't just bad at revision, they're unstable by design. The version drift also echoes the appendix bias finding from the same period, where Claude and GPT showed systematic misalignment with how domain experts weight information. Together these paint a picture of LLM reviewers as fundamentally unreliable gatekeepers, not just in error rates but in consistency across time.

If OpenAI, Google, or Anthropic publish version release notes that explicitly document scoring changes to their models used in peer review, that signals acknowledgment of the problem. If any major venue (ICLR, NeurIPS, ICML) announces they're pinning to specific model versions for 2027 submissions, that's the market responding. If neither happens by Q2 2027, the silence itself becomes the story: providers have no incentive to disclose.

This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error

Coverage behind this analysis

These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.

  1. ·arXiv cs.CL

    Five LLMs fail selective revision test under expert disagreement

    A new audit framework reveals that current LLMs fail to selectively revise decisions when challenged by expert disagreement. Researchers tested five models across 1,989 grounded decisions in medicine, science, and law, finding that none met the mathematical threshold for beneficial revision: they must reverse wrong answers at least nine times more often than correct ones…

    Read Modelwire coverage →Original source ↗

MentionsOpenAI · Google · Anthropic · Claude Opus · Gemini · ICLR

MW

How this coverage is produced

Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Same Papers, Different Verdicts: Tracing Version Drift in LLM Peer Review”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.

More to explore

Five LLMs fail selective revision test under expert disagreement

arXiv cs.CL·

Benchmark reveals gaps in LLM peer review accuracy across error types

arXiv cs.CL·

Chatbot Arena data shows user standards rising faster than model quality decline

arXiv cs.CL·