LLM judges show systematic bias from prior scores in production evaluations

The study isolates a specific cognitive failure mode (anchoring) rather than general model unreliability. The key insight is that LLM judges don't render independent assessments when priors are visible, which contradicts how many teams currently structure evaluation chains.

This connects directly to the Trace Integrity work from August, which flagged that correct answers can mask invalid reasoning paths in production agents. Here we see the inverse problem: evaluation systems themselves become unreliable when context flows through them. Both papers expose gaps between what benchmarks claim and what actually happens in deployment. The anchoring bias finding also echoes the referential uncertainty framework from the dialogue paper, which showed that LLMs don't always update beliefs rationally when encountering new information. The difference is that anchoring bias is systematic and measurable across models, whereas belief revision depends on conversational context.

If teams retrofitting evaluation pipelines to exclude prior scores report measurable improvements in downstream content filtering accuracy or iterative refinement quality within the next six months, that confirms the bias is a production bottleneck rather than a lab artifact. Conversely, if the effect disappears when prior scores are presented in different formats (e.g., as ranges rather than point values), that suggests the bias is presentation-specific and easier to mitigate than the paper implies.