Research·arXiv cs.CL·16h ago

Persona-grounded LLM agents defeat AI detection in online surveys

Researchers have exposed a critical vulnerability in survey-based research: LLM-powered respondents can evade detection when mimicking individual personas rather than generating responses naively. The ASURRE benchmark dataset reveals that while crude AI completion is easily flagged by existing detectors, sophisticated agentic systems that adopt consistent respondent profiles defeat current safeguards. This finding threatens the integrity of academic and commercial surveys across disciplines, forcing researchers to rethink validation protocols and raising questions about the reliability of datasets trained on crowdsourced or online survey data in an era of accessible LLMs.

Modelwire context

Explainer

The critical detail buried in the summary: detectors fail not because LLMs are good at mimicking humans in general, but specifically when they maintain consistent character across multiple survey items. This is a detection problem, not a generation problem.

This connects directly to the reliability concerns surfaced in recent work on LLM confidence and abstention. The Chain-of-Self-Questioning paper from mid-September showed how to make models decline answering when uncertain, addressing fluent but unfounded responses. ASURRE reveals the inverse problem: survey respondents (human or AI) who are confident and consistent are now harder to validate, even when they're synthetic. The accessibility and medical text work also depends on crowdsourced validation data, which this finding calls into question. Together, these papers sketch a pattern where we've built better LLM outputs but weaker verification mechanisms.

If major survey platforms (Qualtrics, SurveyMonkey, academic IRBs) adopt ASURRE-style detection within the next six months, that signals the threat is being taken seriously enough to warrant tooling investment. If they don't, watch whether published survey datasets from 2025 onward start including AI-detection metadata as a standard field, which would indicate the community is shifting toward transparency rather than prevention.

Coverage we drew on

When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsASURRE · LLM

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Towards Detecting AI-Assisted Responses in Online Surveys”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.