Vision models fail at finding hidden objects in images
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Vision-language models struggle asymmetrically with occlusion reasoning, a gap that exposes fundamental limits in how these systems infer hidden information. OccludeBench reveals VLMs achieve only 42.7% accuracy on direct physical cues like shadows versus 62.7% on contextual inference, while humans show no such disparity. This 20-point gap signals that VLMs lack robust spatial reasoning and physical intuition despite their scale. The diagnostic framework OccludeTrace pinpoints failure modes, offering researchers a concrete target for improving robustness in real-world scenarios where partial visibility is common. The finding matters because production VLMs are deployed in safety-critical domains where missing objects carry consequences.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
OccludeBench isolates a specific failure mode: VLMs rely on learned statistical patterns (context) rather than building robust mental models of occluded objects. The 20-point gap between shadow reasoning and contextual inference isn't just a performance dip; it reveals that scale alone doesn't produce physical intuition.
This finding sits directly in a pattern Modelwire has tracked across recent papers. The spatial representation fragility documented in 'When VLMs Destroy What They Know' (October 9) and the predictive spatial reasoning gap from SpaceCast-Bench (October 8) both point to the same underlying problem: VLMs excel at pattern matching on visible information but fail when forced to reason about unobserved state. OccludeBench adds a diagnostic layer by showing that even when physical cues are present (shadows, occlusion edges), models don't use them reliably. The implication is consistent across all three works: current architectures lack the spatial grounding needed for real-world deployment.
If researchers using OccludeTrace identify that fine-tuning on physical-cue reasoning closes the 20-point gap without degrading contextual performance, that would suggest the limitation is training-data driven rather than architectural. If the gap persists after targeted fine-tuning, it signals a deeper representational ceiling that scaling alone won't fix.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Vision-language models fail at predicting unseen spatial outcomes
Vision-language models excel at identifying spatial relationships already present in images, but struggle with predictive spatial reasoning: mentally simulating how scenes transform under intervention and reasoning about hidden outcomes. SpaceCast-Bench, a new 3,862-question evaluation across 182 real-world scenes, exposes this gap by progressing from static perception through local and global prediction tasks. Testing 21 models…
MentionsOccludeBench · OccludeTrace · Vision-language models
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “OccludeBench: Can VLMs Find What's Missing?”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.