Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps

The key distinction buried in this paper is that audio introduces a modality-specific attention signature: the pathological patterns the classifiers learn are not simply borrowed from text-based hallucination detection but are specific to how speech LLMs cross-attend between audio tokens and generated text, which means existing text-only detection tooling does not transfer cleanly.

Hallucination detection has been a consistent thread in recent Modelwire coverage, but mostly in text contexts. The 'Fabricator or dynamic translator' piece from mid-April examined how spurious generation manifests in machine translation and how commercial systems try to manage it, which is a close conceptual neighbor: both papers are asking how to catch a model lying without a ground-truth reference to check against. The difference here is the input modality and the specific signal used, attention maps rather than output uncertainty, which connects loosely to SpecGuard's use of internal model signals for verification rather than external reward models.

The real test is whether these attention-map classifiers generalize beyond Qwen-2-Audio and Voxtral-3B to models with substantially different architectures, particularly encoder-free speech LLMs. If a follow-up evaluation on a third architecture shows comparable detection rates without retraining the classifier, the method has legs; if it requires per-model retraining, its practical value narrows considerably.