Repeated sampling outperforms self-refinement at equal compute cost
A rigorous empirical study challenges the value of sophisticated reasoning techniques like self-refinement and reflexion by showing that simple repeated sampling with majority voting matches or exceeds their performance when token budgets are held constant. Testing seven methods across three model sizes (1.5B to 7B parameters) on mathematics benchmarks, researchers found that elaborate multi-step approaches generate substantially more text without proportional accuracy gains. This finding reshapes how practitioners should think about inference-time compute allocation, suggesting that scaling sampling breadth may deliver better returns than architectural complexity for smaller models.
Modelwire context
Skeptical readThe study tests only mathematics tasks on models up to 7B parameters. The authors don't report whether repeated sampling maintains its edge on reasoning tasks outside math, on larger models, or on domains where reflexion's error-correction logic might matter more than raw sampling breadth.
This connects directly to the evaluation infrastructure concerns raised in the OSReward paper from the same day. Both expose how benchmark choice shapes what we think works: OSReward showed that VLM judges can be unreliable arbiters of agent quality, and this study implicitly asks whether mathematics benchmarks are the right proving ground for inference-time strategies. If evaluation systems are corrupting training signals (as OSReward warns), then a finding that simple sampling beats sophisticated reasoning on math benchmarks might reflect benchmark artifacts rather than genuine capability gaps.
If Wang et al. release results on MATH-500 or other out-of-distribution math splits in the next two months and repeated sampling still wins, that strengthens the claim. If they report results on non-math reasoning tasks (code, logic puzzles, open-ended QA) and reflexion recovers an edge, the finding was domain-specific, not a general principle about inference compute allocation.
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsWang et al. · arXiv
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.