Closing CLIP's modality gap can harm zero-shot accuracy

The paper isolates a specific failure mode: correcting for modality misalignment can concentrate model predictions onto a narrow set of classes, degrading zero-shot accuracy even as the representation gap shrinks. This is not about alignment being unimportant, but about the nonlinear relationship between alignment quality and downstream decision-making.

This finding echoes a recurring theme in recent research: optimization metrics and actual performance often diverge. The BenchMIRT investigation from Hugging Face exposed how benchmarks measure narrow task performance rather than genuine capability, and the SVD-MBR work on metric overfitting showed how systems can game evaluation signals at the expense of unmeasured quality dimensions. Here, we see the same pattern at the representation level: tighter alignment (a clean metric) does not guarantee better predictions (the actual goal). The lesson is consistent across these papers: practitioners must validate that their chosen optimization target actually maps to real-world utility.

If CLIP-based systems that explicitly avoid aggressive modality gap reduction outperform aggressively aligned variants on held-out zero-shot benchmarks (ImageNet, CIFAR, or domain-specific datasets) over the next two quarters, that would confirm this is a production-relevant failure mode rather than a theoretical edge case. Watch whether vision-language model developers publish ablations on alignment strength versus classification performance.