Chest X-ray benchmark audit reveals hidden gaps between protocol and released code
Researchers conducted a forensic audit of a chest-radiograph vision-language model benchmark, tracing discrepancies between the intended experimental protocol and the released artifact. The study uncovered mismatches in prompt bindings, DICOM metadata handling, label extraction, and statistical reproducibility across datasets, model calls, and repository releases. This work exposes a critical gap in medical AI benchmarking: the assumption that benchmark components align is rarely validated post-hoc. For practitioners deploying VLMs in clinical settings, the findings underscore how subtle rendering errors, prompt inconsistencies, and annotation drift can compound through a benchmark pipeline, potentially invalidating downstream model comparisons and clinical claims.62















