New benchmark exposes social reasoning gaps across 20 leading LLMs

The paper's real contribution isn't just measuring social reasoning but surfacing that this capability is measurable at scale and currently far from solved. Most LLM eval work focuses on factuality or instruction-following; social reasoning has been treated as a soft problem without rigorous task decomposition.

This connects directly to the PDE discovery evaluation paper from the same day. Both papers identify a critical gap between what existing benchmarks measure and what actually matters for deployment. Where PDE discovery conflates prediction accuracy with physical plausibility, SoMBench separates mental state inference from norm reasoning from contextual adaptation. The underlying insight is identical: single-metric or surface-level evaluation masks brittleness. For LLMs moving into sustained human interaction, a 72% ceiling on social reasoning suggests current systems may fail outside narrow training conditions, just as ML-discovered physics models fail on new regimes.

If the same 20 models show consistent ranking on SoMBench when tested on held-out human interaction logs (real conversations, not synthetic tasks) within the next six months, the benchmark has predictive validity. If rankings shuffle significantly, the framework is measuring test-taking ability rather than genuine social reasoning capability.