Research·arXiv cs.CL·3d ago

Small VLMs mask uncertainty in degraded images despite internal error signals

Small vision-language models exhibit a critical reliability gap in deployment scenarios: their stated confidence remains artificially stable across degraded images, while internal token probabilities accurately track actual error rates. Testing Qwen2-VL-2B and SmolVLM on compressed, blurred, and poorly lit photos reveals that verbalized uncertainty signals fail to calibrate with performance, creating a safety hazard when systems must decide whether to defer or answer. This finding matters for edge deployment, where resource constraints favor smaller models but uncertainty quantification becomes more important, not less.

Modelwire context

Explainer

The study isolates a failure specific to verbalized uncertainty: small vision-language models can track their own error rates internally (via token probabilities) but fail to communicate that knowledge when asked directly. This is not a general calibration problem but a communication breakdown under realistic degradation.

This connects directly to the taxonomy work from late July, which emphasized that current benchmarks obscure which underlying competencies actually drive performance. Here we see a model that possesses the right internal signal (accurate probability tracking) but lacks the capability to translate that into reliable verbal outputs. The NRC reactor operator study from the same period tested domain knowledge retention in safety-critical settings, but didn't examine whether models could honestly report uncertainty about their own answers. Together these suggest that validation in high-stakes domains requires testing not just accuracy but whether models can reliably communicate confidence boundaries.

If Qwen2-VL-2B and SmolVLM show the same stated-versus-internal gap on the upcoming MMVP benchmark (expected Q3 2026), that confirms this is a systematic property of small vision-language models rather than an artifact of the test setup. If either vendor releases a fine-tuned variant that closes this gap within six months, watch whether the fix generalizes to other degradation types or only patches the specific scenarios tested here.

Coverage we drew on

From Isolated Tasks to Structured Capabilities: A Multilayer Taxonomy for Large Language Models · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsQwen2-VL-2B-Instruct · SmolVLM-Instruct · vision-language models

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.