VLMs fail at spatial reasoning despite strong benchmark scores
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
A new evaluation framework reveals a critical gap in how vision-language models encode spatial information compared to language-based concepts. Researchers introduced EICS and CASP-VL, benchmarks that stress-test whether VLMs maintain consistent internal representations under targeted perturbations. The finding that spatial reasoning degrades substantially below lexical performance, despite strong compositional benchmark scores, exposes a fundamental fragility in multimodal architectures. This matters for practitioners deploying VLMs in tasks requiring reliable scene understanding, and signals that current evaluation metrics may mask serious representation failures.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The critical insight here is not just that VLMs fail at spatial tasks, but that they fail *below their own language priors*. This means the models possess lexical knowledge about spatial relationships (they know 'left of' and 'right of' as concepts) but cannot reliably encode or preserve that knowledge when processing actual images. The perturbation-based testing method (EICS and CASP-VL) is what exposes this gap; standard benchmarks miss it because they test static performance, not robustness.
This finding sits directly atop a cascade of spatial reasoning failures documented over the past two days. SpaceCast-Bench (yesterday) showed VLMs cannot predict how scenes transform under intervention. WOVEN (same day) framed temporal and spatial reasoning as a unified deficit. SAGE (same day) identified attention collapse that breaks visual grounding. What this new work adds is a mechanistic explanation: the models are not just bad at spatial reasoning, they are actively degrading spatial information they already encoded linguistically. This suggests the problem is not missing training data but architectural fragility in how vision and language representations interact.
If the same CASP-VL perturbations applied to models fine-tuned on WOVEN's scene transition data show improved spatial robustness (versus baseline VLMs), that would confirm spatial reasoning is trainable rather than architecturally broken. Conversely, if spatial collapse persists even after WOVEN-style training, the deficit is likely in how multimodal fusion happens at inference time, pointing toward solutions like SAGE's attention intervention rather than better pretraining.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Vision-language models fail at predicting unseen spatial outcomes
Vision-language models excel at identifying spatial relationships already present in images, but struggle with predictive spatial reasoning: mentally simulating how scenes transform under intervention and reasoning about hidden outcomes. SpaceCast-Bench, a new 3,862-question evaluation across 182 real-world scenes, exposes this gap by progressing from static perception through local and global prediction tasks. Testing 21 models…
MentionsVision-Language Models · EICS · CASP-VL · Visual Genome
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.