Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery

The study uses human psycholinguistic ratings, specifically concreteness and imagery scores, as the benchmark rather than task-accuracy proxies, which means the failure mode being measured is semantic in nature: models are not misidentifying objects, they are letting visual salience override lexical judgment in ways humans do not.

This connects directly to the MATCHA paper covered the same day, which argued that embedding-based metrics mask critical model failures by failing to distinguish semantic contradictions. Both papers are pointing at the same underlying problem from different directions: current evaluation frameworks do not adequately capture when models are processing meaning versus processing surface signal. The concreteness study adds a multimodal dimension to that critique, showing that irrelevant visual input can actively degrade semantic reasoning rather than simply failing to help. Together, they suggest the field is accumulating evidence that input richness and semantic fidelity are not the same thing, and that practitioners conflating the two will build systems that fail in non-obvious ways.

If follow-up work shows this degradation pattern holds specifically on abstract or relational concepts (low-concreteness words) but not on high-concreteness nouns, that would give practitioners a concrete filtering heuristic for when to suppress visual context in VLM pipelines.