ConsisGuard: Aligning Safety Deliberation with Policy Enforcement in LLM Guardrails

The paper isolates a failure mode that is distinct from hallucination or general reasoning unfaithfulness: a model can produce a rationale that correctly classifies content as harmful and then output a permissive decision anyway, not because the reasoning was wrong, but because the enforcement step isn't logically bound to it. That's a different problem than getting the reasoning right in the first place.

This connects directly to the evaluation integrity thread running through recent coverage. The RHELM benchmark paper flagged that current evals may overstate production readiness by testing under conditions that don't reflect real deployment complexity. ConsisGuard surfaces a parallel concern: safety systems that pass moderation benchmarks may still fail in production if the benchmark only checks whether the rationale is correct, not whether the final decision follows from it. Both papers are, in different ways, arguing that the thing being measured and the thing that actually matters in deployment are not the same thing.

Watch whether safety benchmark maintainers, particularly those behind widely-used moderation evals, add consistency metrics that score rationale-to-decision entailment separately from classification accuracy. If that doesn't happen within the next two benchmark revision cycles, ConsisGuard's core insight risks staying a paper result rather than shaping how the field actually measures guardrail quality.