Prompt phrasing breaks LLM exam graders despite strong baseline performance

The headline performance parity obscures a critical qualifier: this parity holds only under benign prompting. The study's actual contribution is negative (what breaks models) rather than positive (what makes them work), which reframes the labor-replacement narrative from 'LLMs are ready' to 'LLM evaluation systems are brittle in ways we don't yet understand.'

This connects directly to the CORDIAL calibration work from the same day, which showed that LLM outputs on ordinal scales are systematically miscalibrated despite strong accuracy. Both papers expose a gap between surface-level performance metrics and actual deployment reliability. The current study adds a sharper point: instruction phrasing itself can induce catastrophic failure modes that no amount of post-hoc calibration fixes. The earlier 'JEV vs. LLMs as Rubric Judges' paper also flagged systematic underrating across all automated judges, suggesting the problem runs deeper than any single model or prompt design.

If the researchers test whether the 'never give partial credit' failure mode persists across the same models when the instruction is embedded in a system prompt versus a user message, or when wrapped in a chain-of-thought prefix, that tells us whether this is a parsing fragility or a fundamental reasoning collapse. If the failure doesn't generalize to other ordinal-scale tasks (not just exams), the finding stays narrow; if it does, institutions need to treat LLM graders as adversarially prompt-sensitive systems, not labor replacements.