LLM scoring varies wildly by scale design, complicating social science research
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers have identified a critical blind spot in how LLMs are deployed for social science measurement: the same model produces meaningfully different scores depending on whether you ask it to rate something on a 3-point or 21-point scale, and whether the scale is labeled numerically or semantically. This scale sensitivity undermines comparability across studies and raises questions about the reliability of LLM-based scoring in psychology, political analysis, and content moderation. The work matters because computational social science increasingly outsources measurement to LLMs without standardizing these design choices, potentially invalidating meta-analyses and cross-study conclusions.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper isolates scale design itself (not model capability or prompt wording) as a source of score variance. This means two researchers using the same LLM on identical text can produce incomparable results simply by choosing different rating formats, a problem that existing calibration or prompt-engineering fixes cannot solve.
This connects directly to the PRISM calibration work from October 13th, which proposed correcting systematic LLM judge biases via counterfactual probes. But PRISM assumes the bias is learnable and consistent. This new finding suggests some variance may be structural to the rating interface itself, not a learnable preference. It also echoes the broader pattern from the medical device compliance study (October 15th) and the entity extraction work (October 14th), both of which exposed reliability gaps in deployed LLM systems that no amount of prompt tuning resolves. Scale sensitivity appears to be another class of problem that requires explicit standardization, not just better engineering.
If computational social science consortia (like SSNAP or similar meta-analysis groups) adopt a mandatory scale-format registry for LLM-scored studies within the next six months, that signals the field is treating this as a reproducibility crisis. If they don't, watch whether replication attempts of high-profile LLM-scored studies fail to reproduce effect sizes when scale formats differ from the original.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Researchers calibrate LLM judges using counterfactual probes to align with human preferences
Researchers propose PRISM, a calibration method that corrects systematic scoring biases in LLM judges without human annotation overhead. By probing how model scores shift when content attributes are emphasized or suppressed, the technique learns a correction layer via ridge regression on minimal labeled data. This addresses a critical pain point in AI evaluation: LLM judges…
MentionsLarge language models
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.