Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA
A new study challenges the foundational assumption behind LLM-as-a-Judge systems: that evaluation is inherently easier than generation. Testing across four QA benchmarks, researchers found generation accuracy actually exceeds self-evaluation performance in most cases, with attention analysis showing evaluators spend 3-5x less time reading context and candidate answers than generators do. This finding has immediate implications for anyone building evaluation pipelines or relying on model self-critique for quality control, suggesting the asymmetry may be fundamental rather than a training artifact.62
