LLM essay graders show massive severity bias, halo effects across providers

The critical finding isn't just that LLMs show bias in essay scoring, but that standard validation approaches (agreement metrics, correlation coefficients) actively hide these systematic failures. Pre-registration and psychometric auditing reveal problems that pass conventional benchmarks.

This work belongs to a cluster of papers from late August that challenge how we validate LLM behavior in high-stakes contexts. Like the SUP-MIMIC clinical benchmark (which exposed gaps in how diagnosis reasoning is tested) and the text-to-SQL work on chain ambiguity (which showed single-query accuracy masks real-world failure modes), this paper argues that domain-specific evaluation frameworks beat generic metrics. The essay-scoring audit adds a crucial dimension: it shows that even when LLMs produce reasonable outputs, the underlying rating process can be corrupted by severity drift and halo effects that human rater audits would catch immediately. Educational measurement has had these tools for decades; the gap is that AI evaluation communities haven't adopted them.

If major testing boards (ACT, College Board, or ENEM itself) commission independent audits of their LLM-assisted grading pilots using the same pre-registered rater-effect framework within the next 12 months, this work has moved from academic critique to operational practice. If they don't, the findings remain a methodological warning without institutional uptake.