DevicesWorld benchmark tests LLM agents across mobile, desktop, and IoT

The 6,140-scenario scale is notable, but the more consequential design choice is the explicit inclusion of IoT alongside mobile and desktop, which forces evaluation of agents operating under constrained, asynchronous, and lossy communication conditions that most lab benchmarks quietly exclude.

DevicesWorld arrives alongside a cluster of infrastructure-level work that collectively signals the field moving from single-agent capability to multi-component coordination. The MyAG framework covered the same day addresses the composition and monitoring layer for multi-agent systems, and the two pieces are complementary: MyAG gives builders a structured way to wire agents together, while DevicesWorld gives evaluators a way to stress-test whether those wired systems actually hold up across device boundaries. Separately, the memory management paper from the same batch is relevant here too, since cross-device agents must maintain coherent state across context switches, exactly the failure mode that static memory architectures are shown to struggle with. Together, these three papers sketch an emerging evaluation and infrastructure stack for production-grade orchestration.

Watch whether any of the major agent framework teams, including those building on MyAG-style abstractions, publish DevicesWorld scores within the next two quarters. Adoption by framework builders rather than benchmark authors alone would confirm this is becoming a real evaluation standard rather than a one-off academic contribution.