Convolutional encoders outperform LLMs on mahjong with fraction of parameters
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
A new replication study challenges the assumption that language models excel at complex reasoning tasks. Researchers trained a small convolutional encoder on riichi mahjong decisions using consumer hardware and found it outperformed text-based LLMs on the same training data, achieving 80.6% agreement with expert AI versus 77.8% for the best language model. When scaled to a 4.9M-parameter Transformer, the vision-based approach reached 94.7% agreement and statistically beat 0.8B parameter LLMs at actual gameplay. The finding suggests that for structured decision domains, architectural fit and input representation matter more than scale or language modeling, potentially reshaping how researchers approach game-playing and reasoning benchmarks.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The study isolates a specific failure mode: LLMs aren't bad at reasoning per se, they're bad at reasoning through text when the domain has natural non-textual structure. A small vision encoder outperforms large language models not because vision is superior, but because the problem doesn't need language.
This connects directly to the P2NPCBench work from October 21st, which asked whether LLMs possess genuine reasoning or merely pattern-match surface cues. This mahjong study provides a concrete answer: when you remove the text layer and give models the right input format, reasoning capability emerges at much smaller scale. It also echoes the finding from the MT evaluator study (October 24th) that configuration and representation choices shape model behavior in measurable ways. Together, these papers suggest the bottleneck isn't reasoning capacity but rather how we're encoding problems for models to consume.
If researchers apply the same vision-encoder approach to other structured domains currently benchmarked via text (chess, protein folding, planning problems), and consistently see smaller vision models beat larger LLMs, that confirms this is a general principle about input representation rather than a mahjong-specific artifact. If LLM-based benchmarks remain the standard despite these results, that signals the field values ease of evaluation over accuracy of measurement.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Benchmark exposes whether LLMs reason about computational complexity
Researchers have built P2NPCBench, a benchmark that tests whether large language models can actually reason about computational complexity or merely pattern-match memorized solutions. The work pairs NP-complete problems with their polynomial-time twins, then uses an adversarial multi-agent system called AgentGAN to rewrite problems until frozen code models fail to recognize them, while preserving correctness. This…
MentionsMortal · Riichi Mahjong · Transformer · Convolutional Neural Network
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Text Is the Bottleneck: A Small-Scale Replication of LLM Game Playing in Riichi Mahjong”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.