Study reveals flawed ground truth in hallucination detection benchmarks

The paper doesn't just flag BLEURT as imperfect; it argues that using a similarity metric designed for English paraphrase detection as a proxy for factuality creates systematic label corruption that cascades through downstream benchmarks. The threat isn't to one study but to the comparability of the entire hallucination detection literature.

This extends a pattern exposed across recent work: evaluation frameworks built on flawed proxies. The October 31st safety benchmark audit found that labels leak signal through metadata rather than genuine model behavior. The October 29th data attribution work used synthetic ground truth to isolate which dataset factors break algorithms. Here, the authors are asking the same question one level up: what happens when the labeling process itself is misaligned with what we're trying to measure? Like the RAG abstention study from October 29th, which showed that stronger retrievers surface answers in contexts marked unanswerable, this reveals that improving one component (BLEURT's similarity scores) can mask failures in the actual target (factuality detection).

If papers published after this preprint cite it to retroactively downgrade confidence in their own benchmarks, or if major hallucination detection leaderboards (like those tracking factuality in RAG or summarization) announce label regeneration using alternative factuality metrics within the next two quarters, this finding has shifted practice. If adoption of BLEURT in new hallucination labeling pipelines continues unchanged through Q1 2027, the field has chosen to accept the known flaw.