How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers have exposed a critical vulnerability in vision-language models: their OCR reasoning capabilities degrade sharply under visual corruption, yet this fragility remains largely unmeasured. The new OCR-Robust benchmark systematically evaluates how VLMs handle degraded document images, scene text, charts, and tables, revealing gaps between lab performance and real-world robustness. This matters because production deployments of document AI, receipt scanning, and form processing rely on these models in noisy, low-quality capture environments where visual perturbations are inevitable. The finding signals that current VLM benchmarks may overstate practical reliability.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The benchmark's contribution isn't just identifying that VLMs degrade under noise, which practitioners already suspected, but that it provides a structured taxonomy of perturbation types across document categories, making it possible to compare models on a consistent axis that didn't previously exist.
This fits directly into a pattern Modelwire has been tracking: frontier models performing well on clean benchmarks while failing on structural reliability properties that matter in deployment. The order-sensitivity audit covered in 'Same Evidence, Different Answer' found flip rates of 24-50% across 18 models when input sequences were shuffled, and the mechanism there is analogous here. Both papers are measuring the gap between controlled evaluation and real-world variance, just along different input dimensions. Taken together, they suggest the benchmark ecosystem systematically rewards performance on idealized inputs while leaving robustness to distribution shift unmeasured.
Watch whether OCR1.0 and OCR2.0 model families show meaningfully different degradation curves on the benchmark's scene-text versus structured-document splits. If the gap between those two categories is large, it points to architecture choices rather than training data as the primary driver of fragility.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models
Researchers audited 18 frontier and open-weight multimodal models for order invariance, a foundational reliability property where shuffling input sequences should not change outputs. Using a five-facet framework spanning option ordering, evidence chunking, document ranking, image sequencing, and cross-modal mixing, they found zero models achieved order-invariance, with flip rates between 24-50% per facet. This exposes a…
MentionsVision-Language Models · OCR-Robust · OCR1.0 · OCR2.0
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.