Language models fail consistency tests in imperfect-information games
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
Researchers systematized evaluation of language model agents by scoring decisions at every game tree node rather than just final outcomes. Testing 12B-14B instruct models in imperfect-information games against game-theoretic baselines revealed a critical gap: stated beliefs about opponent state diverge significantly from equilibrium posteriors, and agent actions often contradict those stated beliefs. This behavioral inconsistency exposes a fundamental fragility in how LLMs reason under uncertainty and suggests current models lack coherent internal models of hidden information, a prerequisite for reliable multi-agent reasoning and strategic deployment.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper's core contribution is methodological: by scoring every decision node rather than just final game outcomes, researchers exposed a specific failure mode that aggregate metrics would hide. The finding that stated beliefs contradict both actions and equilibrium posteriors suggests LLMs don't maintain stable hidden-state representations during play, not merely that they play suboptimally.
This connects directly to the mechanistic interpretability work from November 13th on web element hallucination, which showed that internal model states can predict and correct agent failures. Here, the gap isn't between internals and outputs on a single task, but between what the model claims to believe and what its weights actually encode about opponent state. The earlier work on data corruption (November 12th) also fits: both reveal that LLMs rely on surface patterns rather than compositional reasoning under uncertainty. Together, these papers suggest the fragility isn't task-specific but structural to how current models handle hidden information.
If the same 12B-14B models show improved belief-action alignment after fine-tuning on game-theoretic objectives (e.g., via GRPO as tested in the November 15th adaptation study), that would confirm the gap is learnable rather than fundamental to the architecture. If performance remains misaligned even with explicit belief elicitation in the prompt, the problem is likely in the model's representational capacity, not training.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Linear probes predict and correct LLM agent hallucinations on live websites
Mechanistic interpretability research has historically focused on static model behavior, but this work tackles a harder problem: explaining why LLM agents fail in real-world deployment. Using Qwen2.5-7B-Instruct on live websites, researchers discovered that hallucinated web actions can be predicted and corrected via linear probes applied to internal model states, achieving 85% correction rates. The tradeoff…
MentionsLanguage models · CFR+ · Imperfect-information games
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Instrumenting Failure: Scoring Language Model Agents at Every Decision Node in Imperfect-Information Games”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.