Exploiting LLM-as-a-Judge Disposition on Free Text Legal QA via Prompt Optimization

The buried finding here is directional: the researchers didn't just show that prompts affect scores, they showed that optimizing for leniency produces prompts that generalize better across different judge models, which suggests the optimization is capturing something structural about how these models reason rather than overfitting to one judge's quirks.

This connects directly to a cluster of judge-reliability work Modelwire covered in mid-April. The 'Diagnosing LLM Judge Reliability' piece from April 16 found that one-third to two-thirds of documents show logical inconsistencies in pairwise comparisons even when aggregate consistency looks high. That paper treated judge unreliability as a diagnostic problem. This new paper treats it as an optimization target, which is a meaningful shift in framing. The 'Context Over Content' paper from the same week showed judges can be manipulated through stakes signaling. Prompt optimization that systematically tunes judge disposition is a related but distinct attack surface: it doesn't require deceiving the judge about context, it just reshapes what the judge is asked to reward.

Watch whether ProTeGi-optimized prompts hold up when tested against the transitivity-violation diagnostic from the April 16 conformal prediction paper. If lenient prompts also reduce logical inconsistency rates, that strengthens the generalization claim. If they don't, the benchmark gains may reflect calibration drift rather than genuine quality signal.