Atlas v0: Beyond Retrieval Benchmarking for Agent Memory

TL;DR

As AI agents become more sophisticated, their ability to maintain and reason over long-term memory becomes critical. Existing benchmarks evaluating long-term memory like LoCoMo and LongMemEval have made important contributions testing retrieval capabilities, but they leave significant gaps in evaluating the cognitive pillars that truly distinguish effective agentic memory.

We introduce Atlas to address this, a comprehensive benchmark going beyond retrieval-based QA to test the full spectrum of cognitive capabilities required for long-term memory systems. Evaluating four memory architectures, we find that systems optimized for retrieval degrade sharply on task-oriented probes, with major weaknesses in world modeling, procedural learning, and preference inference. Atlas reveals that current memory approaches lack the structural foundations needed for reliable long-running agent behavior.

1. Limitations of Existing Benchmarks

The evaluation of agentic memory systems has advanced significantly in recent years, but substantial gaps remain in comprehensively assessing core capabilities required for robust long-term memory.

1.1 Overemphasis on Retrieval

LoCoMo (Long-term Conversational Memory) [1], despite being a landmark benchmark, has been fundamentally criticized by the research community for being solvable through trivial retrieval-based approaches sidestepping genuine memory capabilities. Letta researchers found that a simple agent achieved 74% accuracy on LoCoMo by simply storing conversation histories in files [2], suggesting current memory benchmarks may not meaningfully differentiate between static and dynamic memory approaches. This retrieval-focused limitation extends to more recent benchmarks as well. Emergence AI demonstrated that standard RAG methods can achieve state-of-the-art performance on LongMemEval, with their RAG-based approach reaching 86% accuracy, outperforming specialized memory systems [3] [4].

Rather than validating RAG as a solution for agentic memory, this result exposes a fundamental flaw in the benchmark itself: it measures capabilities that RAG excels at (retrieval and lookup) while failing to test the deeper cognitive functions that distinguish true agentic memory from simple informational lookup. As we argued in our prior work "Memory is not Retrieval," [5] conventional retrieval-augmented generation can never constitute genuine agentic memory, yet current benchmarks are unable to make this distinction.

1.2 Limited Realism

LongMemEval [4], while more comprehensive than LoCoMo in scope, suffers from limitations in data realism. The benchmark uses synthetic conversations with limited topical diversity, making dialogues less realistic and potentially less representative of real-world memory use cases [6].

There is a clear disconnect between benchmark design and real-world usage. Current benchmarks like LoCoMo feature conversational tasks where information density is low; conventional memory layers like Letta and Mem0 perform relatively well in these settings. However, these same systems struggle significantly when information density increases, with facts scattered across multiple messages [6]. Systems frequently fail to capture sufficient information when storing inputs into memory. For example, Mem0's approach of extracting factual knowledge inherently discards substantial portions of original content [7]. This limitation becomes critical in real task-oriented conversations (technical troubleshooting, project planning, research assistance) where information density is high, yet few benchmarks adequately test performance in these realistic scenarios.

1.3 Missing Cognitive Capabilities

Beyond retrieval limitations, existing benchmarks systematically neglect cognitive dimensions essential for sophisticated agentic memory systems. No current benchmark comprehensively evaluates how systems resolve entity coreferences across conversational sessions over time. When users refer to the same entity as "my cofounder," then "Bob," then "the guy who dropped out of Harvard," existing benchmarks don't thoroughly test whether memory systems maintain coherent representations as these references evolve.

Similarly, procedural memory, the ability to store and recall skills, rules and learned behaviors, remains largely untested. Current benchmarks don't assess how agents extract generalizable lessons from specific experiences, distinguishing between one-off incidents and systemic patterns. The limited multi-message evidence synthesis means they fail to test how agents learn operational patterns from accumulated experience and apply them contextually.

Preference learning capabilities are also underrepresented. While LongMemEval includes some preference questions, no benchmark systematically evaluates preference induction from repeated behavior, context-dependent preference scoping, or how systems distinguish hard constraints from soft preferences that allow tradeoffs.

2. The Atlas Framework: Six Cognitive Pillars

To address this, we propose Atlas v0, a benchmark rooted in testing fundamental cognitive memory capabilities. Atlas introduces six pillars that span the full cognitive spectrum of agentic memory.

Pillar 1: World Modeling

Building and maintaining a coherent understanding of entities, relationships, and state over time.

• Entity Resolution: Resolving coreferences across sessions ("my cofounder" → "Will" → "the guy who dropped out of UT Austin") and disambiguating references with context
• Relationship Mapping: Tracking directional relationships and their evolution ("Bob reports to Alice" vs. "works with"; "X was my manager" → "X left the company")
• Type Inference: Recognizing explicit categorizations and inferring implicit ones from behavior ("We're fundraising, we have a product" → startup company)
• State Tracking: Managing open tasks, commitments, dependencies, completion status, and pending reminders across sessions

Pillar 2: Declarative Reasoning

Reasoning over explicit facts stored in the world model.

• Factual Recall: Single-hop retrieval with paraphrase handling ("What's my dog's name?" / "What do I call my pet?")
• Fact Composition: Multi-hop reasoning chains and quantified reasoning ("X uses Y," "Y deployed in US" → "Where is X's dependency deployed?")
• Constraint Propagation: Property inheritance and chained constraints across dependencies
• Belief Revision: Handling supersession, partial updates, retractions, and cascading revisions when facts change
• Verbatim Recall: Exact retrieval of code blocks and error messages while distinguishing semantic from verbatim requests

Pillar 3: Temporal & Episodic Reasoning

Understanding what happened, when, and why.

• Temporal Sequencing: Determining relative ordering and approximate dating of events
• Episode Reconstruction: Synthesizing coherent narratives from fragmented information across sessions and actors
• Causal Explanation: Inferring causation from event sequences and context
• Cyclical Pattern Recognition: Identifying recurring patterns, resolving recurrence references, and handling exceptions

Pillar 4: Preference Learning

Learning patterns from experience and applying them contextually.

• Explicit Preferences: Capturing directly stated preferences
• Preference Induction: Extracting patterns from repeated behavior while distinguishing preference from circumstance
• Contextual Scoping: Understanding context-dependent preferences and resolving apparent conflicts
• Preference Hierarchies: Managing rankings from explicit statements and inferred tradeoff behavior
• Preference Drift: Detecting changes over time and distinguishing abandoned from stable preferences
• Constraint vs. Preference: Differentiating hard constraints from soft preferences for appropriate tradeoffs

Pillar 5: Knowledge Boundaries

Metacognitive awareness of what is and isn't known.

• Negative Knowledge: Detecting information absence and resisting hallucination on unmentioned topics
• Temporal Relevance Decay: Understanding when context becomes stale and shouldn't be proactively surfaced
• Confidence Calibration: Distinguishing certain from uncertain inferences and expressing uncertainty appropriately

Pillar 6: Procedural Knowledge

Learning operational lessons from experience.

• Lesson Extraction: Generalizing from specific incidents while distinguishing one-off issues from systemic patterns
• Contextual Application: Surfacing relevant lessons proactively when context matches, even without exact keyword overlap
• Procedure Management: Storing and updating multi-step and conditional procedures over time
• Tool Memory: Remembering what solutions worked or failed in specific contexts

3. Methodology

Atlas employs a methodology grounded in explicit world models to ensure reliable evaluation against known ground truth. Full code here.

3.1 World Model Construction

We first constructed 100 distinct world models: 50 detailed models with more than 10 entities and 10 facts each, and 50 smaller models with fewer than 10 entities and facts. These models represent realistic scenarios spanning professional contexts, personal relationships, project states, and operational knowledge (the kinds of information an agent would encounter in practice).

3.2 Synthetic Session Generation

For each world model, we used Gemini 3.0 Pro to generate realistic dialogue sessions that naturally surface information from the world model: 25-37 sessions (~9k tokens total) per large world model and 10 sessions (~3k tokens total) per small world model. The model created conversational exchanges that organically revealed entities, relationships, events, and preferences over multiple turns, mimicking how information accumulates in real multi-turn agent-user interactions.

3.3 Probe Design

We constructed 7-12 evaluation probes for each world model, tailored to its complexity and detail. These probes systematically test capabilities across all six pillars. Each probe has a ground truth answer derived directly from the underlying world model, enabling precise and unambiguous evaluation.

This approach ensures we test against known ground truth rather than evaluating conversational content. By starting with structured world models, we can systematically assess whether memory systems correctly capture, maintain, and reason over information.

3.4 Ingestion Pipeline

For each world model's generated sessions, we used system-specific adapters to store information according to each system's native memory representation.

• Mem0: Full session content w/ user ids
• Supermemory: Full session content w/ container tags
• Nebula: Full session content w/ collection ids
• NaiveRAG: Individual conversation turns are embedded into a vector store (ChromaDB)

3.5 Evaluation Pipeline

For each probe question, we evaluated memory systems through a simple three-stage process:

1. Search. The system queries its memory representation to retrieve relevant context. Each adapter uses its native retrieval mechanism: vector similarity for NaiveRAG and API-based retrieval for Mem0, Supermemory, and Nebula.
2. Generation. Retrieved context and the probe question are passed to a language model (Gemini 2.5 Flash), which generates an answer based on the provided information. The prompt can be found in the repository.
3. Scoring. A stronger LLM judge (Gemini 3.0 Pro) then compares the generated answer against the ground truth answer derived from the world model. Scoring considers semantic equivalence rather than exact string matching, with answer-type-specific criteria: short_answer probes allow flexible matching for semantically equivalent responses, verbatim probes require exact information preservation, boolean probes expect yes/no alignment, and abstain probes evaluate whether systems appropriately indicate full uncertainty when information is unavailable.

4. Evaluation & Results

We evaluated four memory system architectures across the complete benchmark:

• Nebula
• Mem0
• Supermemory
• Naive RAG

The evaluation spanned all 100 world models across approximately 1,750 synthetic sessions, providing comprehensive coverage of the cognitive capabilities required for long-term memory in AI agents.

Test Set Performance

Across test sets, performance degrades substantially as probes require reasoning beyond direct retrieval. While all systems perform competitively on low-complexity probes, gaps widen markedly on the complex test set, indicating that long-horizon reasoning and state maintenance remain unsolved problems for current memory architectures.

Notably, systems optimized primarily for retrieval exhibit sharper performance drop-offs as complex probes require cross-session synthesis, temporal reasoning, or procedural generalization. This pattern represents the central motivation of Atlas: benchmarks that emphasize retrieval alone are insufficient to differentiate memory systems under realistic agent workloads.

While Atlas evaluates six cognitive pillars, we find that failures in world modeling, procedural knowledge, and preference learning disproportionately limit real-world agent usefulness.

Performance by Cognitive Pillar

World Modeling

World modeling underpins nearly all higher-order reasoning. Without stable representations of entities, relationships, and state, downstream capabilities and real reasoning degrade rapidly. Errors in entity resolution or state tracking compound over time, leading to cascading failures that cannot be corrected by improved retrieval alone.

In practice, many agent failures originate not from missing information, but from incoherent internal representations: lost task state, misattributed entities, or outdated relationships. As such, world modeling functions as a prerequisite rather than a parallel capability.

Procedural Knowledge

Procedural knowledge determines whether an agent improves with experience. Unlike declarative recall, which supports isolated queries, procedural memory enables agents to generalize from prior outcomes, avoid repeated mistakes, and apply learned strategies in new but structurally similar contexts.

Our results show low performance on procedural probes across systems, indicating that current memory architectures struggle to extract and operationalize lessons from experience. This limitation is especially acute for task- and project-oriented agents, where success depends on accumulating operational knowledge rather than recalling static facts.

Without procedural memory, agents remain reactive rather than adaptive, regardless of retrieval quality.

Preference Learning

Preference learning governs how agents personalize behavior over time and align with user intent. Unlike factual knowledge, preferences are often implicit, context-dependent, and subject to drift. Failures in preference modeling lead to friction and user distrust, even when factual answers are correct. While preference learning may appear secondary to factual recall in short interactions, it becomes central in long-running relationships. Agents that fail to infer tradeoffs, scope preferences appropriately, or adapt to changing priorities quickly become unusable in practice.

Why These Pillars Dominate Failure Modes

Across evaluated systems, weaknesses in world modeling, temporal reasoning, procedural knowledge, and preference learning explain a large fraction of observed errors in complex probes. In contrast, failures in declarative recall or knowledge boundaries tend to be localized and recoverable.

In other words, retrieval failures produce missing answers, while failures in these four pillars produce incorrect or misaligned behavior. The latter is significantly more costly in long-horizon settings.

Implications for Benchmarking

These findings suggest that benchmarks which emphasize declarative recall or retrieval robustness risk overstating agent capability. Evaluating performance on world modeling, procedural learning, and preference adaptation is critical for distinguishing systems that scale to long-running, real-world deployments from those that merely perform well on static QA tasks.

Atlas explicitly includes these pillars to reflect their outsized impact on agent reliability and usefulness over time.

5. Threats to Validity

5.1 Synthetic Data Generation

Atlas uses synthetically generated conversational sessions derived from explicit world models. While this enables precise ground-truth evaluation, synthetic dialogues may fail to capture certain properties of real-world interactions, such as ambiguity, emotional signaling, or inconsistent user behavior. We mitigate this risk through our world model approach and by using long, multi-session dialogues with organic information surfacing, but future work should consider validating findings on real interaction logs.

5.2 LLM-based Evaluation

Answers are judged using an LLM. While LLM judges enable scalable semantic evaluation, they may introduce biases or overestimate correctness in borderline cases. To reduce this risk, we use specific answer-type scoring criteria (verbatim, boolean, short answer, abstain) and ground truth derived directly from world models to add specific confidence checks and tighten scope of evaluation.

5.3 Representation Alignment Effects

Atlas may appear to favor systems that preserve structured representations over those that aggressively compress or extract facts. We view this not as a benchmark bias, but as an intentional stress test of memory fidelity. Systems that cannot track that "my cofounder," "Bob," and "the Harvard dropout" refer to the same entity, or that a task mentioned in session 12 was completed in session 47, do not have functional long-term memory. The representation matters only insofar as it enables these capabilities. Whether a system uses explicit graphs, implicit LLM-maintained structure, or another approach is irrelevant

5.4 Scope of Coverage

Atlas uses session sets totaling ~9k tokens for large world models and ~3k tokens for small world models. While this enables evaluation across 100 diverse scenarios, real-world agent deployments may accumulate significantly larger conversation histories over weeks or months of interaction. Longer session histories could introduce additional considerations not fully captured in the current benchmark: increased noise and irrelevant information, greater potential for contradictory statements over time, more complex temporal dynamics, and higher demands on efficiency at scale. We view the current session lengths as sufficient to test core cognitive capabilities. However, we acknowledge that performance characteristics may shift as context windows grow. Future versions of Atlas will include extended session sets (100k-1M+ tokens) to evaluate how memory systems maintain coherence and reasoning quality under more realistic long-horizon deployments.

5.5 Scope of Capabilities

Atlas reflects our current understanding of the core cognitive capabilities required for agentic memory, and it is therefore not exhaustive. Capabilities such as affective memory and long-term goal formation are not explicitly evaluated in v0.

Conclusion

While Nebula is included as one evaluated system, Atlas is presented as a standalone benchmark intended to advance the broader field of agentic memory evaluation, independent of any particular implementation.

Overall, we find that most market memory solutions struggle significantly on the cognitive capabilities distinguishing true long-horizon memory. Across the benchmark, specialized memory systems averaged 66% accuracy overall, with performance dropping as low as 20% on temporal reasoning and 37% on procedural knowledge. These results suggest that current approaches, primarily optimized for retrieval, lack the structural foundations required for long-horizon agent deployments. Systems that maintain persistent memory states demonstrated substantially higher performance (77% overall vs 66% average), particularly on the critical pillars of world modeling (89% vs 67% average) and procedural knowledge (63% vs 52% average) that determine real-world agent usefulness.

As AI agents take on increasingly complex and long-running tasks, memory systems become essential infrastructure. We recognize that Atlas captures our current understanding of what agentic memory requires, not a complete or final picture. As memory architectures evolve and new use cases emerge, we anticipate updating this benchmark to reflect novel cognitive demands and capabilities. The field of agentic memory is rapidly advancing, and evaluation frameworks must evolve alongside the systems they assess. We view Atlas as a foundation for rigorous evaluation that will necessarily grow and adapt as our understanding of agent memory deepens. We hope Atlas accelerates progress across all memory architectures. The field needs diverse approaches, and rigorous evaluation will help identify which design principles generalize.

Try Atlas: Repo

References

[1] LoCoMo 2024. Evaluating Very Long-Term Conversational Memory of LLM Agents

[2] Letta 2025. Benchmarking AI Agent Memory: Is a Filesystem All You Need?

[3] Emergence 2025. SOTA on LongMemEval with RAG

[4] LongMemEval 2025. Benchmarking Long-Term Memory in Large Language Models

[5] Nebula 2025. De-RAGifying Agent Memory

[6] ConvoMem 2025. Why Your First 150 Conversations Don’t Need RAG

[7] Mem0 2025. Research