When Context Misleads: Surprisal, Energy and Attention Entropy as Metrics of Coherence Illusions in LLMs

The study's most underreported contribution is the identification of specific attention heads that reliably fail under incoherence conditions, and the demonstration that this failure pattern transfers across experimental setups, suggesting a structural rather than incidental weakness.

This connects directly to the mechanistic attention work covered in 'Does RoPE Prevent or Degrade Retrieval Heads,' published the same day. That paper established that certain attention heads are causally necessary for long-context recall, and their removal collapses performance entirely. The Dutch coherence illusion paper adds a complementary finding: even when retrieval heads remain intact, a separate class of attention heads can be systematically misled by contextual distractors. Together, these two papers suggest the attention mechanism is not a monolith. Different head populations handle different semantic tasks, and robustness failures in one population won't necessarily show up in benchmarks designed to test another. That distinction matters for anyone designing evaluation suites or safety gates for production systems.

The key test is whether the specific failing attention heads identified in Dutch models have functional analogs in English or multilingual models. If replication studies on larger, widely deployed models find the same head-level failure signatures, the vulnerability is architectural and not a Dutch-specific artifact.