Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agents
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
A critical gap in how we measure agent safety has emerged: distinguishing between genuine risk avoidance and mere incapacity. PhoneSafety, a new 700-instance benchmark spanning 130+ mobile apps, isolates this problem by forcing a three-way classification at risky moments: safe action taken, unsafe action taken, or complete failure. This matters because current evaluations conflate these outcomes, masking whether an agent's harmlessness stems from learned judgment or architectural limitation. For practitioners building phone-use systems, the distinction determines whether to retrain, redesign, or debug. The benchmark exposes a fundamental weakness in existing safety methodology that has likely inflated confidence in deployed agents.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The deeper provocation here is epistemological: every safety score produced by prior phone-agent evaluations may be measuring the wrong thing entirely, because a model that refuses to act due to confusion looks identical, in aggregate metrics, to one that refuses due to principled caution.
This is largely disconnected from recent activity in our archive, as we have no prior coverage of mobile agent safety benchmarks or phone-use agent research to anchor against. The work belongs to a broader cluster of agent reliability research, sitting alongside ongoing debates about how capability and alignment interact in agentic settings. The core tension PhoneSafety surfaces, that harmlessness and safety are not synonymous, has been implicit in alignment discourse for years but rarely operationalized at the task level. Without prior Modelwire coverage to triangulate against, readers should treat this as an entry point into that conversation rather than a development in a thread we have been tracking.
Watch whether major phone-agent frameworks (Google's Android agent work or any derivative of AppAgent) publish PhoneSafety scores within the next two quarters. If leading labs adopt the three-way classification in their own evals, the benchmark has traction; if it stays confined to citation counts, the methodology may be sound but practically ignored.
This interpretation is generated from the summary above and available source metadata. Our methodology · Report an error
MentionsPhoneSafety
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.