Multilingual models fail to recognize names across scripts, study finds
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Multilingual language models exhibit severe identity recognition failures when names shift from Latin to non-Latin scripts, even when semantic context remains constant. Testing across Qwen2.5-7B and four competitors reveals same-person accuracy collapses from 35% to 8.5% under Devanagari prompts, while false-positive rates stay stable. The effect replicates across language families using Malayalam data, suggesting tokenization improvements alone cannot bridge the gap. This finding exposes a fundamental architectural vulnerability in how current LLMs handle script-dependent entity disambiguation, with direct implications for multilingual deployment reliability and bias auditing in production systems.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper isolates a failure that persists even when tokenization is adequate: models lose entity identity not because they can't represent non-Latin scripts, but because their internal representations of the same entity diverge based on script alone. This suggests the problem lives in embedding space or attention mechanisms, not the tokenizer.
This connects directly to our October 17 coverage on Indic script tokenizer failures, but inverts the diagnosis. While that story showed tokenizers emit 17x more tokens for Kannada, this new work implies tokenization improvements won't solve the core problem. The entity matching collapse also echoes the October 21 finding that non-native input amplifies model failures, suggesting script-switching acts as a form of linguistic perturbation that destabilizes internal representations across multiple failure modes.
If Qwen or competing vendors release updated versions with script-aware embedding initialization or cross-script contrastive training within the next six months, and accuracy recovers to above 25% on Devanagari entity matching, that confirms the vulnerability is fixable at the architectural level. If accuracy remains below 15% despite tokenizer upgrades, the problem is deeper than current mitigation strategies assume.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
TinyLlama's tokenizer fails on Indic scripts, exposing efficiency measurement flaws
A new arXiv study exposes a critical blind spot in how small language model efficiency is measured and marketed. Researchers found that TinyLlama-1.1B, despite claims of multilingual capability, degrades catastrophically on Indic scripts, emitting 17x more tokens for Kannada than English and falling back to raw UTF-8 bytes for Odia over 93% of the time.…
MentionsQwen2.5-7B · Devanagari · Malayalam · Marathi
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Same Entity, Different Script: Diagnosing Cross-Script Entity Matching Failures in Language Models”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.