From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?

The key buried detail is that the LLM advantage here is specifically about predicting *aggregate* subgroup opinions, not individual ones. That distinction matters enormously: a model can be systematically wrong about any given person while still being a better estimator of what a group believes on average, because its errors cancel out in ways that inconsistent human annotators' errors do not.

This connects directly to the reliability questions raised in 'Diagnosing LLM Judge Reliability' from April 16, which found that aggregate consistency metrics (~96%) masked serious per-instance logical failures. That paper and this one are essentially describing the same phenomenon from opposite sides: aggregate statistics can flatter models even when individual-level behavior is unreliable. The 'DiscoTrace' paper from the same week adds another angle, showing LLMs lack the rhetorical variety humans bring to answers, which suggests their annotation advantage may be narrow and task-specific rather than general.

The real test is whether the structural conditions the researchers identify as 'common in real-world scenarios' hold up when applied to politically or culturally contested annotation tasks, where subgroup opinion variance is high and model training data is likely skewed toward majority perspectives.