Skip to content
Modelwire
Subscribe

Vision-language models fail on perception, not reasoning, study finds

Source published ·Modelwire updated

Original coverage: arXiv cs.CL ↗·How Modelwire adds context

The development

A new study challenges the prevailing theory that vision-language models fail at theory-of-mind tasks due to reasoning deficits. Researchers demonstrate that performance gaps stem primarily from perceptual limitations rather than conceptual ones, using a dissociation protocol that isolates visual fidelity from reasoning complexity. The finding reframes the bottleneck in multimodal AI: models can reason about false beliefs when given text, but struggle to extract the necessary information from visual inputs. This shifts the research agenda away from adding reasoning modules toward improving visual scene understanding, with implications for how teams prioritize architectural improvements in vision-language systems.

Modelwire’s AI-generated summary of coverage from arXiv cs.CL.

Modelwire analysis

Explainer

Our AI-generated reading of the wider context and the next developments to watch.

The study isolates perception as the binding constraint by holding reasoning constant (via text) and varying visual fidelity, not just by comparing models on mixed tasks. This dissociation protocol is the methodological move that lets them claim causation rather than correlation.

This finding sits in a cluster of recent work decomposing where vision-language models actually fail. Last month's counting study (Seeing vs. Counting) showed arithmetic reasoning was the bottleneck on diagrams despite accurate visual input. This new work inverts that: when reasoning is held constant via text, perception becomes the limiting factor. Together they suggest the field has been chasing the wrong fixes. The crochet execution work from today also reinforces this pattern: models produce plausible outputs that fail under real-world grounding, hinting that intermediate representations (what the model actually extracts from images) are weaker than benchmark scores suggest.

If MuMA-ToM's Perceptual Contribution Ratio metric correlates with downstream performance on embodied tasks (like the crochet benchmark) when applied to the same models, that validates perception as the unified bottleneck. If it doesn't, the metric is a local artifact and the field still lacks a unified diagnostic.

This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error

Coverage behind this analysis

These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.

  1. ·arXiv cs.CL

    Multimodal LLMs fail at counting, not seeing, on diagram tasks

    Researchers isolated a critical failure mode in multimodal LLMs by decoupling visual perception from computational reasoning. Using paired counting tasks of varying algorithmic complexity on identical diagrams, they found that Gemini, Gemma, and Qwen3-VL models typically read small images accurately but fail at the counting step itself, with error rates driven more by answer magnitude…

    Read Modelwire coverage →Original source ↗

MentionsHeider-Simmel · MuMA-ToM · vision-language models · Perceptual Contribution Ratio

MW

How this coverage is produced

Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Perception, Not Reasoning, Limits Visual Theory of Mind”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.

More to explore

Hybrid planning beats end-to-end VLMs on occluded vision tasks

arXiv cs.CL·

Multimodal LLMs fail at counting, not seeing, on diagram tasks

Vision-language models hide cross-image reasoning in hidden states

arXiv cs.CL·
Vision-language models fail on perception, not reasoning, study finds · Modelwire