How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring

The deeper problem here is circularity: the field has been using attack-success rates to rank defenses and compare models, but if the judges producing those rates disagree with each other by a factor of ten on recall, then the entire leaderboard logic collapses. The paper doesn't just flag noise, it suggests that published safety comparisons may be ordering models incorrectly.

This connects directly to a pattern Modelwire has been tracking around measurement infrastructure failing to keep pace with capability claims. The 'Fault of Our Stars' piece from the same day made a structurally identical argument in a different domain: numeric scores used as ground truth turn out to be noisy labels, and models trained or evaluated against them inherit that noise silently. The same logic applies here. Where that paper flagged weak labels in sentiment datasets, this paper flags weak labels in safety evaluation, and both point to a shared blind spot: practitioners trusting proxy metrics without validating the proxies themselves.

Watch whether HarmBench or any major safety leaderboard issues a methodology update that standardizes judge selection within the next six months. If they don't, published attack-success rates will continue to be compared across papers that used incompatible judges, making cross-study conclusions unreliable.