Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA

The attention analysis is the part worth sitting with: evaluators allocating 3-5x less processing to the very context they're supposed to judge suggests this isn't a fixable calibration problem but something closer to a structural limitation in how evaluation is computed internally.

The mechanistic framing here connects directly to the Vision-Default, Prior-Override paper covered the same day, which found that only 2.5-4.8% of attention heads govern knowledge override in VLMs. Both papers are doing the same kind of work: using attention patterns to explain why a model fails at a task that looks easy from the outside. Together they reinforce a broader pattern in recent interpretability research where sparse, localized attention behavior turns out to explain capability gaps that benchmark numbers alone obscure. That matters practically because anyone relying on LLM-as-a-Judge for QA pipelines, including the truth-fusion workflows described in the Single and Multi Truth Data Fusion paper from the same day, is now building on an evaluator that may be systematically under-reading the evidence it's supposed to weigh.

Watch whether LoRA fine-tuning on evaluation-specific attention patterns closes the generation-evaluation gap on MuSiQue specifically. If it does not, that confirms the asymmetry is architectural rather than a training data artifact.