Cross-survey transfer exposes limits of LLM respondent simulation
Researchers propose cross-survey transfer as a stricter evaluation method for silicon sampling, where LLMs predict individual respondent answers across different question sets rather than matching aggregate distributions. Testing on Taiwan election data with models ranging 27B to 120B parameters reveals zero-shot LLMs achieve only 52% accuracy, suggesting current models struggle with coherent individual-level prediction despite apparent pattern-matching success. This work challenges the validity of prior silicon sampling claims and establishes a more rigorous benchmark for assessing whether LLMs genuinely simulate human respondents or merely reproduce statistical artifacts.
Modelwire context
Skeptical readThe key omission from prior silicon sampling work: aggregate-level pattern matching does not imply individual-level coherence. This paper shows that models can reproduce statistical distributions while failing catastrophically at predicting single respondent trajectories, meaning earlier validation may have measured statistical mimicry rather than genuine respondent simulation.
This connects directly to the data leakage and evaluation methodology concerns surfaced in recent coverage. The RF drone benchmarks paper (early July) exposed how standard evaluation splits can mask overfitting through near-duplicate data; this Taiwan study reveals a parallel problem in social science simulation where aggregate metrics hide individual-level failure. Both papers share a pattern: published results rely on evaluation frameworks that are too coarse to catch model brittleness. The groupthink piece from MIT Technology Review also hints at this: models cluster around predictable outputs, which could explain why silicon sampling appears to work at population scale but collapses when forced to maintain coherence across question sets.
If researchers apply cross-survey transfer to prior silicon sampling datasets (especially those cited as validation for LLM-as-respondent claims) and report accuracy below 55%, that confirms the earlier work overstated model capability. Conversely, if any model family reaches 70%+ accuracy on this benchmark within the next six months, watch whether that correlates with architectural changes (longer context, different training objectives) rather than just scale.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsTaiwan Election and Democratization Study · TEDS 2024
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Silicon Sampling via Cross-Survey Transfer”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.