Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study

The more pointed finding is that OCR-specialized models, systems built explicitly for text recognition, are outperformed by general-purpose frontier models on degraded Devanagari, which suggests the specialization is narrower than advertised and likely optimized around Latin and CJK script test sets.

This connects directly to a pattern Modelwire has been tracking across multiple papers from this same window. The KrishokChat paper (story 4) flagged how English-centric benchmarks mask real performance gaps for the billions of users whose languages sit outside dominant training distributions. That work focused on Bengali agricultural text; this benchmark makes the same structural argument but targets vision systems rather than language models. Both papers are pointing at the same upstream problem: evaluation suites that reward performance on high-resource scripts while leaving low-resource script failure invisible until a stress test forces it into view. The diffusion LLM evaluation piece (story 1) adds a third angle, showing that benchmark rankings shift with methodology, which means the OCR field's confidence in its own leaderboards may be doubly fragile.

Watch whether Qwen or DeepSeek release updated model cards that include Devanagari degradation benchmarks within the next two quarters. If they do, it signals the research community's pressure is reaching training pipeline decisions; if they don't, this benchmark risks becoming a citation without consequence.