Multimodal LLMs fail at counting, not seeing, on diagram tasks
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers isolated a critical failure mode in multimodal LLMs by decoupling visual perception from computational reasoning. Using paired counting tasks of varying algorithmic complexity on identical diagrams, they found that Gemini, Gemma, and Qwen3-VL models typically read small images accurately but fail at the counting step itself, with error rates driven more by answer magnitude than problem complexity class. This decomposition reveals that vision-language model weaknesses on diagram tasks stem less from image understanding and more from arithmetic reasoning under uncertainty, suggesting that improving counting robustness may require different interventions than enhancing visual encoding.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The study's key contribution isn't just identifying counting failures, but demonstrating that vision-language models read small diagrams correctly yet still produce wrong answers. This reverses where practitioners should focus remediation efforts: the bottleneck is arithmetic reasoning under uncertainty, not visual encoding.
This connects directly to the November 3rd construction safety study, which found that Qwen2.5-VL and InternVL3.5 correctly recognized objects and spatial relationships yet still made systematic errors. Both papers reveal the same pattern: multimodal models can extract accurate visual information but fail at the reasoning step that follows. The current work adds precision by showing the failure isn't stochastic noise but driven by answer magnitude, suggesting the models are struggling with numerical confidence calibration rather than basic arithmetic. This also echoes the November 2nd benchmarking audit, which exposed how scoring rules can mask genuine capability gaps. Here, the gap between perception and reasoning has been hiding in plain sight because researchers weren't decomposing the task.
If the same three models (Gemini, Gemma, Qwen3-VL) show improved counting accuracy after fine-tuning on arithmetic reasoning tasks without retraining the vision encoder, that confirms the diagnosis. Conversely, if visual encoder improvements alone don't move counting performance, the finding holds. Watch whether Google or Alibaba acknowledge this decomposition in their next model release notes or safety audits.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.LG
Vision-language models prioritize learned patterns over spatial reality in safety tasks
Vision-language models deployed for construction safety are generating false alarms at rates that undermine their practical utility, but not because they misidentify objects or spatial relationships. A new study of Qwen2.5-VL and InternVL3.5 reveals both models correctly recognize workers and excavators and encode proximity cues, yet still flag hazards incorrectly roughly one-quarter of the time.…
MentionsGemini · Gemma · Qwen3-VL · Google · Alibaba
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Seeing vs. Counting: Complexity Twins Show Where Multimodal LLMs Fail on Diagrams”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.