Frontier models fail at copying due to positional encoding bias

Illustration accompanying: Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

Frontier LLMs fail at a task humans find trivial: copying input text within their context window. Researchers trace the failure to how Transformers encode position information, which biases models toward pattern matching rather than precise token location tracking. Their solution, 2D-RoPE, reframes text as a grid with row and column coordinates, reducing copying to fixed-offset retrieval. This work exposes a fundamental architectural limitation in current models and suggests positional encoding design directly shapes what reasoning shortcuts models learn, with implications for how future architectures should handle spatial reasoning and long-range dependencies.

Modelwire context

Explainer

The failure mode here is more specific than 'LLMs hallucinate': models aren't losing track of meaning, they're losing track of location, which means the problem is geometric rather than semantic. 2D-RoPE doesn't patch the model's knowledge, it changes how the model represents where tokens live in space.

This connects directly to the rate-utility frontiers paper from the same day ('Rate-Utility Frontiers for Language Encodings'), which asks how encoding choices shape what information survives into the model's latent space. That paper treats encoding as a compression question; this paper treats it as a spatial reasoning question. Together they suggest the field is converging on a harder question: not just what information gets encoded, but what geometric structure that encoding imposes on downstream computation. The shortcut-reliance work on spoken English auto-markers ('Controlling Implicit Shortcut Reliance') also rhymes here, since both papers show Transformers defaulting to statistical patterns when precise structural tracking would be more correct.

Watch whether 2D-RoPE gets adopted in any publicly announced architecture within the next two model generations from labs already using RoPE variants, such as Meta or Mistral. Adoption there would confirm the fix is practical at scale, not just a controlled benchmark result.

Coverage we drew on

Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsTransformer · RoPE · 2D-RoPE · LLM

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.