Letta Leaderboard: Benchmarking LLMs on Agentic Memory Recent

Published: May 29, 2025 | Original Post

---

A comprehensive benchmark suite evaluating LLM performance on agentic memory management.

Memory Hierarchy

• Core memory -- Information within the agent's context window, organized by memory blocks
• Archival memory -- External context management beyond the immediate context window

Benchmark Components

Memory Read

Tests retrieving facts from core memory and archival memory search effectiveness.

Memory Write

Simulated conversations test whether agents appropriately save important information. Retention measured through subsequent Q&A after chat history removal.

Memory Update

Contradictory information tests whether agents recognize and update outdated memory.

Top Performers

• Claude Sonnet 4 (Extended Thinking), GPT-4.1, and GPT-4o lead overall
• For cost-conscious applications: Gemini 2.5 Flash and GPT-4o-mini provide solid memory performance at a fraction of the cost

Methodology

Uses GPT-4.1 for grading responses against ground-truth answers following OpenAI's SimpleQA methodology, with penalties for unnecessary memory operations.