Model version swaps flip ICLR acceptance verdicts by 79 points
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Frontier LLM providers are silently rolling out model updates that dramatically shift evaluation outcomes, undermining the reliability of peer review at major venues. Researchers auditing Claude Opus and Gemini versions across ICLR 2027 submissions found acceptance rates swinging by 43 to 79 percentage points between model iterations, with no papers improving under newer versions. The instability persists even when controlling for review text, suggesting fundamental scoring drift. This exposes a structural vulnerability in AI-driven academic gatekeeping: benchmark scores treated as stable measurements are actually hostage to provider release cycles, creating invisible bias in which research gets published.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
Analyst takeOur AI-generated reading of the wider context and the next developments to watch.
The critical detail buried in the summary: acceptance rates swing wildly not because reviewers change their minds, but because the scoring model itself drifts between versions. No papers improve under newer versions, suggesting this isn't random noise but directional bias baked into provider release cycles.
This connects directly to the audit framework from mid-November showing LLM peer reviewers fail to selectively revise decisions when challenged (the expert disagreement study). That work exposed a gap between apparent responsiveness and actual reasoning. This new finding suggests the gap runs deeper: the reviewers aren't just bad at revision, they're unstable by design. The version drift also echoes the appendix bias finding from the same period, where Claude and GPT showed systematic misalignment with how domain experts weight information. Together these paint a picture of LLM reviewers as fundamentally unreliable gatekeepers, not just in error rates but in consistency across time.
If OpenAI, Google, or Anthropic publish version release notes that explicitly document scoring changes to their models used in peer review, that signals acknowledgment of the problem. If any major venue (ICLR, NeurIPS, ICML) announces they're pinning to specific model versions for 2027 submissions, that's the market responding. If neither happens by Q2 2027, the silence itself becomes the story: providers have no incentive to disclose.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Five LLMs fail selective revision test under expert disagreement
A new audit framework reveals that current LLMs fail to selectively revise decisions when challenged by expert disagreement. Researchers tested five models across 1,989 grounded decisions in medicine, science, and law, finding that none met the mathematical threshold for beneficial revision: they must reverse wrong answers at least nine times more often than correct ones…
MentionsOpenAI · Google · Anthropic · Claude Opus · Gemini · ICLR
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Same Papers, Different Verdicts: Tracing Version Drift in LLM Peer Review”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.