MEME: Multi-entity & Evolving Memory Evaluation
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
A new benchmark reveals a critical failure mode in LLM-based agents operating across multiple sessions. MEME exposes that memory systems collapse when reasoning about dependencies between stored facts, achieving near-zero accuracy on cascade and absence tasks despite strong static retrieval. The finding matters because production agents increasingly need persistent memory across conversations, yet current architectures and optimization strategies fail to bridge this gap, suggesting fundamental architectural rethinking is needed before deployment in real-world multi-turn environments.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The critical detail buried in the benchmark design is that MEME specifically isolates cascade and absence reasoning, meaning agents must track how one stored fact invalidates or depends on another across sessions. This is categorically different from retrieval quality, and the near-zero accuracy scores suggest current memory architectures were never actually designed to handle this class of problem.
Two related benchmarks published the same day make this cluster worth treating as a signal rather than coincidence. LongMemEval-V2 targets a similar gap, asking whether agents internalize environment-specific knowledge over extended interactions rather than just retrieving it on demand. Together, these two efforts suggest the evaluation community is converging on relational and temporal reasoning as the next frontier for agent memory, not storage capacity or retrieval speed. The 'Learning, Fast and Slow' paper from the same day adds relevant architecture context, since its dual-timescale framing directly addresses how parameter updates and in-context adaptation interact, which is precisely the tension MEME exposes in memory systems.
Watch whether any of the major agent memory frameworks (MemGPT, Zep, or comparable open projects) publish MEME scores within the next two quarters. If none do, that absence itself signals the benchmark found something practitioners would rather not measure publicly.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues
Researchers have released LongMemEval-V2, a benchmark designed to measure how well AI agent memory systems retain and apply environment-specific knowledge over extended interactions. The work addresses a gap in existing evaluations, which typically measure short-term recall or task completion rather than whether agents genuinely internalize interface patterns, state transitions, and failure modes needed to operate…
MentionsMEME · LLM-based agents
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.