JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers have exposed a critical blind spot in LLM-as-a-judge systems: their verdicts shift dramatically when prompts are semantically reworded, even though the underlying task remains identical. The JudgeSense benchmark quantifies this instability across nine judge models, revealing that factuality evaluations suffer from a polarity-inversion artifact that tanks consistency to 0.63 before correction. This finding matters because automated evaluation is becoming infrastructure for model development and deployment. If judges can't maintain stable decisions across prompt variations, their use in production pipelines introduces hidden bias and unreliability into the feedback loop that shapes which models get shipped.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The more pointed finding isn't just that judges are inconsistent, it's that factuality tasks specifically produce polarity inversions, meaning a judge can flip from 'correct' to 'incorrect' on the same answer depending on how the question is framed. That's not noise, that's a systematic directional bias baked into how these models process evaluation prompts.
This is largely disconnected from recent activity in our archive, as we have no prior coverage to anchor it to. It belongs to a growing body of work interrogating whether automated evaluation can be trusted as a substitute for human judgment. The practical stakes are high because LLM-as-a-judge is now embedded in RLHF pipelines, red-teaming workflows, and leaderboard scoring. A consistency score of 0.63 before correction, in a component that is supposed to be objective infrastructure, is the kind of number that should make teams audit what their eval stack is actually measuring.
Watch whether major eval frameworks like HELM or EleutherAI's lm-evaluation-harness incorporate prompt-variation stress tests within the next two release cycles. If they do, JudgeSense will have shifted from academic critique to operational standard; if they don't, this stays a warning that practitioners can safely ignore.
This interpretation is generated from the summary above and available source metadata. Our methodology · Report an error
MentionsJudgeSense · Judge Sensitivity Score
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.