Research·arXiv cs.LG·2d ago

Medical vision models fail generalization test across TB screening datasets

Medical vision-language models routinely fail to generalize across real-world deployment conditions, according to a systematic audit of three specialized models against general-purpose baselines. Researchers tested BioMedCLIP, CheXficient, and MedSigLIP on tuberculosis screening across four chest X-ray datasets, finding that benchmark rankings collapse when cohorts, prompts, or disease prevalence shift. The work exposes a critical gap between controlled evaluation and clinical robustness, suggesting that current benchmarking practices mask fragility in high-stakes medical AI. This challenges the field's confidence in model selection for regulated deployment.

Modelwire context

Skeptical read

The paper doesn't establish that benchmarking practices are fundamentally broken, only that three specific models fail to generalize across datasets and disease prevalence shifts. The critical omission: whether the same benchmark collapse occurs with general-purpose vision models tested identically, or whether this is specific to vision-language architectures trained on limited medical data.

This echoes a pattern from the dysarthric speech work (September 18) which found that pooled, heterogeneous training fails when conditions diverge. But that paper proposed a solution: per-aetiology separation. The chest X-ray audit stops at diagnosis without testing whether disease-specific model variants or stratified evaluation protocols recover the benchmark signal. It's a problem statement without a remedy, whereas the speech work moved to architecture.

If the authors release ablations showing that retraining any of the three models on a single dataset (Montgomery or VinDr-CXR) restores benchmark ranking stability, the finding becomes 'models need domain-specific tuning,' not 'benchmarks are misleading.' If ranking remains unstable even on single-dataset retraining, that's a genuine red flag about model capacity or label quality. The distinction determines whether this is a deployment problem (solvable) or a measurement problem (harder).

Coverage we drew on

Per-Aetiology Contrastive Severity Embeddings with Phonological Pseudo-Labelling for Multilingual Dysarthric Speech · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsBioMedCLIP · CheXficient · MedSigLIP · OpenCLIP · Montgomery · VinDr-CXR

Read full story at arXiv cs.LG →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screening”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.