Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers audited 18 frontier and open-weight multimodal models for order invariance, a foundational reliability property where shuffling input sequences should not change outputs. Using a five-facet framework spanning option ordering, evidence chunking, document ranking, image sequencing, and cross-modal mixing, they found zero models achieved order-invariance, with flip rates between 24-50% per facet. This exposes a critical gap between benchmark performance and real-world robustness that emerging AI safety guidelines now demand. The finding signals that current MLLM evaluation misses systematic brittleness that could undermine deployment in high-stakes settings where input presentation varies.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper's most underreported finding is structural: the five-facet Facet-Probe framework reveals that brittleness is not concentrated in one modality or input type but is distributed evenly across text, image, and cross-modal combinations, which means there is no safe subset of multimodal inputs that escapes the problem.
This connects directly to the voice AI reliability paper covered the same day ('Real-Time Voice AI Hears but Does Not Listen'), which found that production systems detect signals they then fail to act on. Both papers are documenting the same underlying gap: models that pass capability evaluations while failing on consistency and reliability properties that deployment actually requires. The order-sensitivity finding extends that concern from a single modality to the full multimodal stack, and together the two papers suggest that current benchmarking practice is systematically blind to a class of behavioral failures that only surface when input presentation varies.
Watch whether any of the 18 audited model providers, particularly Gemini given its explicit naming, publish targeted robustness evaluations or training updates that address order sensitivity within the next two quarters. Silence from vendors after a named audit is itself informative.
This interpretation is generated from the summary above and available source metadata. Our methodology · Report an error
MentionsGemini · Facet-Probe · multimodal large language models
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.