Benchmark exposes LLM theory-of-mind gap beyond game rules

The critical insight isn't that models fail at Avalon, but that they fail in a specific way: they memorize rules and social facts without actually modeling other agents' mental states. This distinction between factual knowledge and genuine belief reasoning is what the benchmark isolates.

This work sits directly alongside the social judgment benchmarking from August 10th (the persona ratings study), which found that LLMs show internal consistency on social assessments but may not be substituting for human judgment in meaningful ways. Avalon-ToM-Bench goes deeper: it suggests the consistency itself might be surface-level pattern matching rather than genuine social reasoning. Both papers flag the same underlying problem from different angles: we've been measuring what models can retrieve or rank, not what they can actually infer about other minds. The PragMatch work on vision-language models reinforces this pattern, showing that multimodal systems also mistake shortcut cues for genuine reasoning.

If the same 28 models show better performance when given explicit access to other players' previous statements (removing the need to infer intent), that confirms the gap is reasoning-specific rather than architectural. Watch whether follow-up work on the benchmark shows whether chain-of-thought prompting or in-context examples of belief modeling close the gap, or whether the failure persists even with scaffolding.