Memory Is Not Retrieval

The future of AI requires persistent agents: systems that accumulate institutional knowledge, maintain coherent world models over time, and can be trusted with consequential decisions. This requires memory as infrastructure: a substrate that supports both decision traces (the why behind actions) and reasoning (world modeling, temporal logic, procedural knowledge, preferences).

Current memory systems built on retrieval cannot provide this foundation: long contexts degrade performance [1], models ignore information in the middle [2], and vector retrieval has fundamental geometric limits [3]. We argue that memory must be abstracted as a persistent state, with retrieval serving only as a projection of that state into a bounded context window.

1. What Persistent Agents Need From Memory

The shift from AI assistants to AI colleagues requires agents that persist across time, accumulating institutional knowledge, maintaining coherent world models, and being trusted with consequential decisions.

Doing this requires two classes of capability.

A. Decision tracing

Right now, enterprises capture what happened but lose why. A discount gets approved, an exception gets granted, an escalation gets routed, but the rationale disappears. The service incidents that justified the discount, the precedent from last quarter's similar deal, the Slack thread where a VP approved the exception: none get systematically stored.

As Foundation Capital describes in its piece on context graphs [4], this missing layer of decision traces is where real operational knowledge lives. Production agents need to capture:

• Provenance: who decided, under which policy version
• Precedent: what similar cases existed and how they were resolved
• Exceptions: when rules were overridden and why
• Approval chains: the human judgments that shaped outcomes

When this information persists, precedent becomes searchable and exceptions evolve into policy instead of repeatedly re-tried edge cases.

B. Reasoning (the cognitive layer)

Beyond decision traces, persistent agents need memory that supports structured reasoning. These are the cognitive capabilities that separate a competent colleague from a lookup table:

• World modeling: maintaining coherent beliefs about entities and relations
• Temporal reasoning: what was true at decision time vs. what's true now
• Declarative/procedural knowledge: facts and patterns
• Preference learning: what the user values and why
• Knowledge boundaries: what the agent knows vs. doesn't know

Retrieval can’t do this. You can't do temporal reasoning by fetching chunks. You can't maintain a coherent world model when "Alice is a designer" and "Alice is an engineer" compete as separate embeddings. You need a structured state that evolves with explicit semantics.

2. Invariants of Persistent Memory

From these use cases, we can extract a set of invariants. Any memory system supporting persistent agents must have the following properties.

1. Persistence across sessions : Alice tells you her preferences in January. They should still be there in June.
2. Auditability : When the agent says "Alice prefers async," you should be able to trace where that belief came from and when.
3. Canonicalization : Entities, beliefs, and policies must have stable canonical identities so that updates, provenance, and precedent attach to the same underlying object over time.
4. State consolidation : Source data persists, but the agent can't condition on unbounded history. Memory must consolidate beliefs into a coherent, queryable state that fits inference budgets.
5. Precise updates : When Alice says "I switched from engineering to product," that should replace the old fact, not sit alongside it as a contradiction.
6. Retrieval as view, not mechanism : In most systems, retrieval is memory: the agent "knows" only what gets fetched in top-k. This forces the LLM to reconstruct state from noisy, overlapping retrievals rather than querying a deterministic structure. Facts should exist as structured state that can be queried precisely, with retrieval being just a projection into context.

Most systems fail at (4), (5), or (6). Vector retrieval stores every fact independently, forcing competing beliefs into top-k. RAG appends indefinitely until context budgets collapse. Knowledge graphs handle updates but require constant manual maintenance and lack automatic abstraction.

When memory has these five properties, it becomes a queryable artifact that other system instances can build on. But you can't build a context graph on retrieval. Decision traces require provenance (who decided, under which policy version), temporal validity (what was true at decision time), and contradiction resolution (when two rules conflict, which took precedence). That's persistence, auditability, precise updates, and retrieval as projection, the invariants above.

3. Why Retrieval Can't Get There

Most agent memory is RAG: store conversation as chunks, embed, retrieve top-k, append to prompt. This works for QA over documents, but it structurally cannot support decision traces or reasoning.

Three failure modes in production

1. Contradictions (blocks reasoning) Your RAG store has "Alice is a web designer" from January and "Alice is a software engineer" from March. Which one gets retrieved? Both? The model has to guess which is current, often using unreliable heuristics [5]. You can't do world modeling when beliefs compete.

2. Context pollution (blocks decision traces) Shoving more "relevant" chunks into the prompt doesn't make agents smarter. Research shows models get less reliable as context grows, even on simple tasks [1]. They also ignore information buried in the middle of long prompts [2]. Decision context gets buried in noise. More tokens ≠ better memory.

3. Embedding ceiling (blocks scale) Even with perfect training, a single-vector similarity retriever cannot represent all task-relevant subsets of a growing memory store [3] because of the geometric limits of fixed-dimensional embeddings. At scale, "Alice likes pizza" and "Alice's friend likes pizza" become mathematically inseparable, causing recall to degrade. The agent has to reconstruct state from noisy, overlapping retrievals rather than querying deterministic structure.

The abstraction is wrong

Retrieval framing treats memory as an episodic query primitive:

$y = f(q, \text{Retrieve}(q, D))$

This means the agent "knows" only what gets fetched in top-k for any given query. But for reasoning and decision traces, memory must be a queryable state that exists independently of any single retrieval, with explicit update semantics, temporal validity, and contradiction resolution.

Why not just store memory in model weights?

Weights are powerful for generalization, but they're a poor substrate for controlled, deployable memory updates. Continual weight updates are prone to interference and forgetting, reflecting the stability-plasticity tradeoff in continual learning [6]. And even targeted "model editing" can introduce unintended side effects on general capabilities as edits accumulate [7]. Explicit, externalized state naturally supports the need for memory to be auditable, reversible, and updatable at the level of individual beliefs.

4. The Abstraction Shift: Stateful Conditioning

We propose a new paradigm: stateful conditioning.

Stateful conditioning treats memory as an explicit state that persists across interactions. Each new observation applies a precise update. Retrieval becomes a budgeted projection of state, not the mechanism of memory itself.

Formally, contrary to retrieval-as-memory, stateful conditioning treats the world model as an explicit memory state ($S_t$) updated over time:

$S_{t+1} = U(S_t, o_t) \quad y_t = f(o_t | S_t)$

Where:

• $o_t$ is a new observation (messages, tool outputs, outcomes)
• $U$ is a bounded update operator
• $S_t$ persists independently of any single query

Retrieval becomes a budgeted projection of state, not the mechanism of memory itself.

This abstraction enables behaviors retrieval-centric systems struggle with:

• Explicit state transitions with provenance: "Alice is a web designer" and later "Alice is a software engineer" don't become two competing chunks that the model arbitrarily guesses over at inference time. Instead, they form an explicit state transition (supersession) with provenance and recency.
• Temporal validity: Ensures the system can recall facts known at decision time, not just what is true at present.
• Contradiction resolution: When two beliefs conflict, the system has explicit semantics for which one governs (recency, confidence, policy hierarchy), not arbitrary similarity scores.
• Deterministic queries: Instead of reconstructing state from noisy retrievals, the agent queries structured state directly.

5. Nebula: Infrastructure for Persistent Agents

We built Nebula, a memory substrate designed around stateful conditioning and budgeted projection. Here’s how it works:

5.1 Conceptual Memory

Nebula treats memory as an evolving conceptual state encoding what the agent believes about the world:

• Entities and relations: Alice, Project X, "Alice works_on Project X"
• Ontology: Explicitly typed entities/relations plus definitions for metrics and business rules (e.g., Customer → Orders; “active customer” = purchased in last 90 days; “VIP” = spend > $5k) [8].
• Provenance: Where did this belief come from? (user statement, tool output, inference)
• Temporal validity: When was this learned? Is it still current?
• Confidence: How certain is this belief?

This creates a knowledge state the agent can reason over. It unifies three types of memory into one substrate:

• Facts (semantic/declarative): “Alice is a product manager.”
• Events (episodic/evidence): “On March 12, Alice experienced an Azure outage.”
• Preferences/Policies (action-shaping): “Alice prefers async updates over Slack DMs.”

5.2 Bounded Memory Operations

Raw experience is immutable. New observations append to an event log (messages, tool outputs, outcomes). Nebula’s memory is the evolving state derived from that log: beliefs are consolidated, updated, and invalidated through bounded operations that preserve provenance and history.

Nebula provides bounded operations that maintain memory coherence:

• ADD : introduce a new belief with provenance
• MERGE : deduplicate equivalent beliefs and entities
• SUPERSEDE : replace outdated beliefs while preserving history
• DEPRECATE : lower confidence / invalidate without erasing provenance

Every change is auditable (what changed and why) and reversible.

memory_id = nebula.store_memory({
    "collection_id": "demo",
    "content": "Alice and Bob are making a startup Nebula together",
    "metadata": {"topic": "AI", "difficulty": "intermediate"}
})
# ADD belief "Alice and Bob are cofounders"

# New observation
client.store_memory({
    "memory_id": memory_id,  # Add to existing memory
    "collection_id": "demo",
    "content": "Alice and Bob split"
})
# DEPRECATE "Alice and Bob are cofounders"

5.3 Two-path pipeline

Nebula handles memory on two timescales:

• Test-time: Produces a recall view for inference and applies lightweight state updates that must complete within latency constraints.
• Sleep-time: Asynchronously consolidates memory (dedupes, conceptualizes, and stabilizes timelines) without blocking inference (see [9]).

Memory improves between interactions, not just during them. Consolidation runs continuously so the agent's world model stays coherent as history accumulates.

5.4 Adaptive recall

Recall is a budgeted projection of state:

• Narrow factual queries surface minimal, high-signal memory
• Multi-entity or multi-step queries expand to the smallest supporting view required

results = nebula.search(
    query="When did I want to check my groceries?",
    collection_ids=["personal-planning"]
)
# Factual query, surfaces high-signal facts and utterances

results = nebula.search(
    query="Who was the person my boss said to talk to?",
    collection_ids=["personal-planning"]
)
# Multi-relational query, recall expands to surface more related facts

Closing

The future of AI agents isn't better retrieval. It's a queryable state.

Today’s agents can’t maintain decision traces: the precedents, exceptions, and reasoning that make memory actionable. This is an infrastructure problem. You can’t build a decision graph on top of text search. Agents built on retrieval can't capture decisions as durable artifacts because retrieval has no concept of state. They also can't reason over a structured state. World modeling, temporal logic, and procedural knowledge all require memory that evolves, not static chunks.

Stateful conditioning changes what's possible. An agent doesn't retrieve "discount approved." It knows the incidents that justified it, the precedent it followed, who approved the exception, and which policy version governed the decision.

We're building the memory layer the agent era requires. If you're building agents that need to remember, evolve, and reason across time, Nebula gives you the infrastructure to do it right.

References

[1] Chroma Research. Context Rot.

[2] Liu et al., 2023. Lost in the Middle: How Language Models Use Long Contexts

[3] Weller et al., 2025. On the Theoretical Limitations of Embedding-Based Retrieval.

[4] Foundation Capital, 2025. AI’s trillion-dollar opportunity: Context graphs

[5] Wan et al., 2024. What Evidence Do Language Models Find Convincing?

[6] Kirkpatrick et al., 2016. Overcoming Catastrophic Forgetting in Neural Networks.

[7] Gu et al., 2024. Model Editing at Scale Leads to Unintended Side Effects.

[8] Chen, 2025. why ontology for text-to-sql?

[9] Letta, 2025. Sleep-time Compute: Beyond Inference Scaling at Test-time.