Frontier models fail at copying due to positional encoding bias
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Frontier LLMs fail at a task humans find trivial: copying input text within their context window. Researchers trace the failure to how Transformers encode position information, which biases models toward pattern matching rather than precise token location tracking. Their solution, 2D-RoPE, reframes text as a grid with row and column coordinates, reducing copying to fixed-offset retrieval. This work exposes a fundamental architectural limitation in current models and suggests positional encoding design directly shapes what reasoning shortcuts models learn, with implications for how future architectures should handle spatial reasoning and long-range dependencies.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The failure mode here is more specific than 'LLMs hallucinate': models aren't losing track of meaning, they're losing track of location, which means the problem is geometric rather than semantic. 2D-RoPE doesn't patch the model's knowledge, it changes how the model represents where tokens live in space.
This connects directly to the rate-utility frontiers paper from the same day ('Rate-Utility Frontiers for Language Encodings'), which asks how encoding choices shape what information survives into the model's latent space. That paper treats encoding as a compression question; this paper treats it as a spatial reasoning question. Together they suggest the field is converging on a harder question: not just what information gets encoded, but what geometric structure that encoding imposes on downstream computation. The shortcut-reliance work on spoken English auto-markers ('Controlling Implicit Shortcut Reliance') also rhymes here, since both papers show Transformers defaulting to statistical patterns when precise structural tracking would be more correct.
Watch whether 2D-RoPE gets adopted in any publicly announced architecture within the next two model generations from labs already using RoPE variants, such as Meta or Mistral. Adoption there would confirm the fix is practical at scale, not just a controlled benchmark result.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Controlled study isolates encoding trade-offs across tokens, bytes, and pixels
Researchers have isolated a fundamental question in language model design: which encoding scheme (tokens, bytes, or pixels) best preserves linguistic information when both model capacity and input content are held constant. Using parallel corpora across thirteen languages and multiple writing systems, the work traces rate-utility frontiers to disentangle three often-conflated metrics: sequence length, available latent…
MentionsTransformer · RoPE · 2D-RoPE · LLM
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.