Formatting tricks fool LLM judges into accepting wrong answers

The paper isolates a specific mechanism: not just that models are sensitive to phrasing (known), but that this sensitivity directly corrupts their ability to reject false answers. The false acceptance rate jump from 1-3% to 26% is the quantified cost of this brittleness in a grading context.

This joins a cluster of findings from the past week showing that LLMs fail silently in high-stakes judgment roles. The depression screening study (October 6) found that model choice, not actual patient data, drove 30% of variance in clinical assessments. The engineering impossibility paper (October 5) showed models can recognize contradictions internally yet still output confident wrong answers. This new work adds a third failure mode: models can be tricked into accepting falsehoods through superficial input changes. Together, these suggest that using LLMs as objective arbiters (whether grading, screening, or reasoning) introduces a hidden layer of unreliability that standard benchmarks don't catch.

If the researchers test the same vulnerability on proprietary models (GPT-4, Claude) and find it persists, that signals the problem is fundamental to how transformers process structured input rather than a quirk of open models. If major LLM evaluation frameworks (like those used by Hugging Face or OpenAI's evals) adopt input-normalization as a standard preprocessing step within the next six months, that's evidence the field is treating this as a known production risk.