SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems

The deeper problem SocialGrid surfaces is a measurement validity issue: when agents fail at navigation and spatial planning, you can't isolate whether they're also failing at social deduction, because the two failure modes are entangled in the same score. The benchmark may be measuring motor-cognitive bottlenecks more than theory-of-mind.

This lands one day after CoopEval (covered April 16), which tested LLM agents in social dilemmas like prisoner's dilemma and found models defaulting to defection rather than cooperation. Together, the two papers sketch a consistent picture: LLMs struggle with multi-agent social reasoning across very different task framings, whether the setting is abstract game theory or embodied grid navigation. The difference is that CoopEval's failure mode is clean and attributable, while SocialGrid's confounded design makes it harder to know what exactly is breaking. That distinction matters for anyone trying to use benchmark results to guide model development rather than just rank models.

Watch whether follow-up work disentangles navigation competence from social reasoning by holding movement ability constant, perhaps through oracle-pathing ablations. If scores remain low under those conditions, the social intelligence deficit is real; if they recover substantially, SocialGrid is measuring the wrong thing.