Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks

The paper's most pointed finding is not just that coverage is incomplete, but that the gaps cluster around categories like service disruption and model internals, precisely the attack surfaces that are hardest to probe through behavioral testing alone. That specificity matters more than the headline percentage.

This connects directly to two threads already running on Modelwire. The 'Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demands' piece from the same day formalized what it calls the audit gap, arguing that behavioral evaluation cannot inspect latent representations or long-horizon planning. This taxonomy paper is essentially empirical evidence for that same structural failure, showing the gap is not hypothetical but measurable across six real benchmarks. Meanwhile, 'MetaBackdoor: Exploiting Positional Encoding as a Backdoor Attack Surface in LLMs' demonstrated an attack class that targets architectural components rather than inputs, exactly the kind of vector that falls outside current benchmark coverage by design. Together, these three papers describe a coherent problem: the field is testing for threats it already knows how to describe, while novel attack surfaces accumulate outside the frame.

Watch whether HarmBench or AgentDojo publish a formal response or updated coverage roadmap within the next two quarters. If neither benchmark incorporates the taxonomy's uncovered categories into a versioned release, that confirms the audit gap is structural rather than a temporary lag.