LLMs struggle to tell stories out of order, new benchmark shows
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers have built the first systematic benchmark for evaluating whether large language models can generate narratives with nonlinear structure, where story events are told out of chronological order. Testing 19 models against 83 published works reveals a consistent gap: LLMs default to chronological narration even when given explicit instructions to deviate, matching human-authored nonlinear storytelling only about half the time. This work exposes a meaningful limitation in current model capabilities around narrative sophistication and creative constraint-following, with implications for content generation, creative writing tools, and understanding how well LLMs grasp complex structural reasoning.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The benchmark itself is novel, but the finding is less surprising: the real insight is that explicit instructions don't override LLMs' default chronological bias. This suggests the limitation runs deeper than training data distribution and points toward something structural in how these models reason about sequence and causality.
This connects directly to the temporal reasoning gap exposed in ChronoCulture (October 14), which found that LLMs fail to track how concepts shift across time. Where ChronoCulture measured static cultural attitudes versus temporal evolution, this work measures whether models can even construct deliberately scrambled timelines. Both reveal that LLMs struggle with non-default temporal reasoning, even when given explicit constraints. The pattern also echoes findings from Latent Taxonomies (October 11), which showed that structure emerges naturally during pretraining without supervision. Here, structure (chronology) appears so deeply baked in that models resist overriding it, suggesting temporal ordering may be a learned inductive bias that's harder to suppress than to acquire.
If the same 19 models show significantly better performance on a held-out test set of nonlinear narratives that were explicitly marked with temporal markers (flashback labels, timestamps) during training, that would suggest the gap is solvable through data augmentation rather than architectural. If performance stays flat, it points toward a reasoning limitation that fine-tuning alone won't fix.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
LLMs struggle to track cultural attitudes across time
Researchers have exposed a fundamental limitation in how LLMs understand cultural change. ChronoCulture, a benchmark of 10,360 temporal survey pairs spanning 72 countries, reveals that while models can capture static cultural attitudes, they fail to track how those attitudes shift over time. Model predictions across different years cluster too closely together, suggesting LLMs lack genuine…
MentionsLLMs · GPT · Claude · Llama
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.