The dream is: the model remembers what you said before and draws meaning across it over time. Not just recall, but interpretation, narrative, the kind of memory that makes a conversation feel continuous and cumulative across months or years. Today, you can achieve an illusion of this dream. For days, or weeks if you're lucky. Until the LLM starts forgetting and the illusion breaks.
Why does this happen?
As your conversation history grows, the memory system must decide what to capture, how to represent it, and what to surface on any given conversation turn. Every one of those decisions is lossy, opinionated, and non-deterministic. Over time, either the corpus of information becomes too large to reliably search, or what the system remembers starts to drift from what was actually said due to repeated summarization. The model forgets because the system either can't hold a complete picture, or the picture becomes distorted. In an ideal world, LLMs would have perfect historic context on the conversation turns that matter. Infinite attention across every word you've ever exchanged, with none of the cost or latency that would actually entail. Since that's not possible, every memory system is an attempt to approximate it. Each with its own drawbacks. There are ultimately only two ways to preserve information from a conversation: 1. Raw — original messages, stored verbatim 2. Derived — summaries, narratives, structured extractions Every memory system is choosing a position on this spectrum. And neither extreme works. Raw is lossless but inert. A pile of transcripts isn't understanding. The information is all there, but nothing is connected, prioritized, or interpreted. It's just buried in the source material. Derived is compact and usable, but repeated derivation drifts from the source the way a photocopy of a photocopy degrades. You don't lose the information all at once. You lose it gradually, and can't tell exactly when it stopped being accurate.
Won't infinite context solve this?
This is the most natural objection. Context windows keep getting bigger. Won't they eventually get big enough that we can just skip the memory system entirely and feed in the full history? Not anytime soon. For two reasons: 1. Cost. Even if you could fit two years of conversation history into a context window, you'd be paying to process all of it on every single turn. The economics are brutal and they scale linearly with history. No consumer product survives that margin structure. 2. Degradation. Models get worse as the context window fills. Attention drops on information in the middle, overall reasoning quality declines, instruction following gets sloppier. You're paying more for worse performance. Infinite context is just the extreme version of the raw path. And we've already established why raw alone doesn't work.
The evaluation paradox
To know if a memory system is working, you need ground truth. But for real conversational memory spanning months or years, the ground truth is the entire history, which is larger than any context window and larger than any human can reasonably annotate. Benchmarks like LongMemEval can test needle-in-haystack retrieval, but retrieval alone isn't memory. Memory is what happens when facts change, when old context gets superseded, when the significance of a conversation only becomes clear weeks later. The right answer depends on the full arc of the relationship, and the arc is always in motion. No benchmark captures arcs. Synthetic datasets can't replicate these real-world examples at scale. The conversations lose coherence long before they reach realistic length, and even if they didn't, nobody can confirm ground truth for a synthetic relationship that evolved across a million tokens. That's why every new memory approach you see land is a different set of trade-offs dressed up as a solution. Nobody can actually prove theirs works, because any judge you'd use to evaluate the full history has the same context limitations as the system it's judging.
Where this leaves us today
The dream from the top of this piece, a model that remembers what you said and draws meaning across it over time, requires solving both sides of the raw/derived tradeoff simultaneously. Perfect preservation and perfect interpretation. Every current approach sacrifices one for the other. This isn't a criticism of the people building these systems, it's an honest description of the constraint they're working within. Compression is lossy. Retrieval is imperfect. And the thing we actually want, meaning that accumulates and evolves, might be the hardest thing to formalize in a system that runs on pattern matching over tokens. Memory for LLMs remains unsolved not because nobody's tried hard enough, but because the problem is very, very hard to solve. We're getting closer. But we're not there.
Deep Dive: How memory systems are built
"There are no solutions. There are only trade-offs." — Thomas Sowell
Every memory system is a composition of choices across a set of axes. Different products pick different paths, but the axes themselves are stable: 1. What gets stored 2. When derivation happens 3. What triggers a write 4. Where it gets stored 5. How it gets retrieved 6. Post-retrieval processing 7. When retrieval happens 8. Who is doing the curating 9. Forgetting policy When a new memory solution lands, you can lay it on this map and see exactly which choices it made and which ones it's punting on.
1. What gets stored
Memory systems hold onto either raw material, derived material, or some mix. Raw: Derived: This is not an exhaustive list - there can be infinite types of derivations, but there are just some common ones.
2. When derivation happens
Derived artifacts have to be produced somewhere, and the timing is its own design decision.
3. What triggers a write
Every memory system has to decide when to capture something at all. Write-triggering is upstream of everything else. If you write the wrong things, no amount of clever retrieval will save you.
4. Where it gets stored
The storage backend constrains everything downstream. Most real systems use more than one. A common pattern is filesystem or document DB for the source of truth, vector DB for retrieval, and sometimes a graph DB on top for relationship traversal.
5. How it gets retrieved
Storage backend constrains retrieval strategy. Each strategy has a characteristic strength. Semantic search is good at "find me things conceptually like this." Full-text is good at exact phrases and proper nouns. Graph traversal is good at "what does the system know about this entity and everything connected to it." Filesystem navigation is good when the model is expected to actively explore.
6. Post-retrieval processing
Once you have candidates, you usually want to narrow further. Post-processing is where a lot of the perceived quality of a memory system actually comes from. Cheap retrieval plus smart re-ranking often beats expensive retrieval alone.
7. When retrieval happens
Three modes. These have very different failure modes. Always-injected pollutes context with irrelevant history. Hook-driven covers passive awareness but is expensive and can make the model perform memory rather than have it. Tool-driven respects the model's judgment but the model doesn't know what it doesn't know, so it often fails to fetch when it should.
8. Who is doing the curating
At every decision point in a memory system, something is making a choice. Who or what is it? Worth tracking as its own dimension because the cost, quality, and accountability profile of each curator is different. Systems that put the main model in charge of curation pay for quality on every turn. Systems that put cheap models in charge pay less but get sloppier decisions. Systems that lean on the user are accurate but add friction.
9. Forgetting policy
Every memory system has a forgetting policy, whether or not the designer chose one. The question isn't whether to forget, it's how. What gets forgotten: How forgetting propagates: Forgetting in a memory system isn't a single delete operation. If you stored raw turns and derived summaries from them, deleting the raw turns doesn't delete the summaries. If you extracted facts into a graph, deleting the source conversation leaves the facts orphaned. Real forgetting requires either tracking provenance (so you can cascade deletes) or periodically re-deriving everything from a smaller raw corpus, which is expensive. When forgetting happens: Forgetting is structurally hard for the same reason the rest of memory is: you don't know at write time what'll matter later, and you don't know at delete time what'll matter later either. Forgetting too aggressively means losing context the user wanted preserved. Forgetting too conservatively means accumulating an inaccurate model of the user that gets harder to correct over time. There's no right setting, only trade-offs.
Common failure modes
Every memory system fails. The question is how, and whether the failure is recoverable. These are the patterns that show up most often in practice.
New session starts with no awareness of previous ones. The user is back to zero every time.
The model misidentifies or merges distinct entities during derivation. Two people with the same name become one, or categories bleed into each other.
The model jumps to conclusions and encodes exaggerated or incorrect interpretations as facts. Without careful prompting, it fills gaps with plausible-sounding fabrications.
Chained summarizations compound small errors. Each derivation is slightly lossy, and the losses accumulate. After enough rounds, the derived memory diverges from what was actually said.
The system surfaces semantically similar but contextually wrong memories. Embeddings are close, but the meaning is different.
Old, heavily-referenced memories crowd out recent ones. The system keeps surfacing outdated context because it was discussed more frequently.
Retrieval only finds what matches the current query's framing. Relevant memories stored under a different topic or emotional register are invisible.
When summaries replace raw turns, specific details vanish. The compression is lossy in ways that destroy the most useful information.
The system states a "memory" with full confidence but there's no way to trace it back to what was actually said. The user can't tell if this was stated, inferred, or hallucinated.
The system's responses are always colored by what it already knows about you. Sometimes that helps. But sometimes you want an uncolored take. As you can see, building a working memory system is incredibly complex and gets harder in every dimension as the information grows. I hope this map helps you see the full design space, pick the right trade-offs for your product, and know which failure modes you're signing up for. | https://x.com/chrysb/status/2043020014035570784 |
- Session amnesia
- Entity confusion
- Over-inference
- Derivation drift
- Retrieval misfire
- Stale context dominance
- Selective retrieval bias
- Compaction information loss
- Confidence without provenance
- Memory-induced bias