How AI Memory Systems Store Context
Open a chat interface. Type something. Get a response. Reference a point from three exchanges back, and the model handles it without missing a beat. The whole thing feels…
Open a chat interface. Type something. Get a response. Reference a point from three exchanges back, and the model handles it without missing a beat. The whole thing feels continuous, like talking to someone who has actually been listening.
The model isn't remembering, though. Not the way you are.
What feels like continuity is the model re-reading the entire conversation each time you send a message. It isn't holding a thread the way your brain does, maintaining some ambient awareness of what came before. It's processing a document that contains the conversation so far, generating a response, then waiting. When the session ends, that document is gone.
This distinction is more consequential than it first seems. Humans forget gradually, imprecisely, in ways shaped by salience and repetition. AI systems forget completely and immediately, by default, unless someone builds infrastructure to prevent it. The forgetting isn't a bug in the conventional sense; it's a consequence of how the architecture works. And every engineering decision downstream flows from that gap between how AI memory feels and how it actually functions.
What "Context" and "Memory" Actually Mean in an LLM
These two words get conflated constantly, and the conflation becomes costly the moment you're building something real.
Context is everything the model can touch during a single interaction: the conversation so far, any background material you've injected, the system prompt shaping its behavior. Situational. Transient. Cleared when the session ends. Memory is the broader category, the system's capacity to retain and utilize information either across sessions or from training. One is a scratchpad; the other is closer to a filing system.
The analogy researchers reach for is worth borrowing: the context window is RAM, fast and immediately accessible, wiped when the process closes. Parametric weights, the knowledge baked into the model during training, are closer to a hard drive. Available, but not easily rewritten.
What makes that framing useful isn't its elegance. It's that when you're deciding whether to stuff more history into the prompt or reach for a retrieval layer, you're navigating exactly this terrain. The question isn't purely technical; it's about what kind of knowledge the system needs and how durable that knowledge has to be. Conflating context and memory at the vocabulary stage produces systems that conflate them architecturally, which is a substantially more expensive mistake to fix.
The Memory Taxonomy: From Working Memory to Parametric Weights
Researchers surveying this field have converged on a set of memory types drawn from cognitive science. The taxonomy is useful not because it's exhaustive, but because it surfaces decisions that otherwise get made by instinct and regretted at scale.
Working memory is the context window: fast, immediately useful, gone when the session closes.
Episodic memory stores specific past interactions with temporal and contextual metadata. This is what allows a correction a user made in session three to surface in session seven. Without it, every session starts from scratch regardless of what came before.
Semantic memory covers structured factual knowledge: user preferences, product catalogs, organizational policies. Many architectures split this into user-specific and environment-specific categories because retrieval strategies differ in ways that matter under load.
Procedural memory is the one that gets skipped in most product conversations, and skipping it is a mistake worth examining. In LangChain's LangMem library, procedural memory allows agents to rewrite their own operating instructions over time. No simpler system has a real equivalent. The CoALA framework treats it as a formal category, and if you're building autonomous agents and you ignore it, you'll eventually run into exactly the wall it was designed to address.
Parametric memory is knowledge internalized into model weights during training: zero retrieval latency, enormous breadth, fixed at training time. You can't update it without retraining or fine-tuning, and that process carries real risk. Updating one region of the weight space degrades performance in adjacent regions. The technical term is catastrophic forgetting, which is about as apt a name as machine learning has ever produced.
Non-parametric memory inverts this: externalized storage in RAG systems and vector stores. Flexible, updatable, and entirely dependent on retrieval quality. The flexibility you gain, you pay for in pipeline complexity.
The Context Window: Scope, Mechanics, and the Cost of Making It Bigger
The context window sets the token budget the model can consider at once. GPT-4o runs a 128,000-token window, which sounds generous until you're simultaneously loading long documents, multi-session history, retrieval results, system instructions, and tool outputs. The ceiling appears faster than expected, and when it does, the failure mode is often subtle rather than obvious.
The underlying mechanism is self-attention. In a Transformer architecture, attention lets every token in a sequence relate to every other token simultaneously, producing coherence across long passages. The cost: compute scales quadratically with sequence length. Double the tokens and you've roughly quadrupled the processing load.
Scaling the window also introduces what the literature calls the "Lost-in-the-Middle" phenomenon. Models systematically underweight information positioned toward the center of long inputs; the beginning and end receive disproportionate attention. A 128,000-token window doesn't give you 128,000 tokens of equally reliable recall. It gives you a gradient of reliability that degrades toward the middle, which is a fundamentally different thing from what most teams assume they're getting when they see that number.
The context window, treated as the entire memory system, functions like L1 cache with no L2, no paging, no eviction policy. Research measuring 4.45 billion tokens found 21.8% structural waste and 84.4x amplification of stale content as a direct consequence of this design gap. Extending the window without building the surrounding hierarchy doesn't resolve the problem; it defers it while making the underlying structural inefficiency more expensive.
External Memory: How RAG Turns Vector Stores into a Retrieval Layer
Retrieval-Augmented Generation is the most widely deployed response to what the context window can't handle. The workflow is conceptually simple: embed your knowledge-base documents as numerical vectors, store them, retrieve the most semantically relevant chunks at query time, inject them into context before generation begins. Getting it to work well in production is considerably less simple.
Embeddings aren't keyword indexes. They're dense numerical representations of meaning, positioned in high-dimensional space such that semantically related concepts cluster geometrically. "Refund" and "return policy" retrieve as related without sharing any surface-level text. That property is what makes RAG useful for natural-language queries against large corpora, and it's also what makes debugging retrieval failures counterintuitive: the relationship between a query and a failed retrieval isn't visible the way a missed keyword would be.
The practical advantages are real. Only the retrieved chunks need to fit in the window, not the entire knowledge base. The system can incorporate content that postdates the model's training cutoff. Domain-specific knowledge becomes available without retraining.
The ceiling is retrieval quality, and this is where teams consistently underinvest. The retrieval layer surfaces the wrong chunks, or misses the right ones, and the model generates from a flawed foundation regardless of how capable it is otherwise. Teams ship RAG systems that technically work and practically fail because the embedding model was an afterthought, the chunking strategy was arbitrary, the metadata schema was sparse. These aren't implementation details. They're the system's actual epistemic foundation, and treating them as secondary is the most common expensive mistake made in this space.
Advanced RAG research is pushing toward adaptive retrieval, graph-augmented retrieval, multimodal grounding, and hybrid search pipelines. Common vector store backends include Pinecone, Weaviate, and Chroma, each with different latency profiles and scaling characteristics. Which backend you choose matters considerably less than whether you've thought carefully about retrieval strategy first.
Layered Architectures: How Modern Systems Stack These Mechanisms Together
No serious production system relies on a single memory type. The architectures that hold up under real load combine multiple mechanisms under coordinated management, each layer handling what it's best suited for.
A common stack: an episodic store for conversation history and temporal metadata, typically Redis or MongoDB; a knowledge base for factual content in a vector database; an action history layer logging structured records of tool calls and outcomes. The coordination between these layers is where most of the interesting engineering actually lives.
MemGPT, now developed under the Letta project, takes the most explicitly architectural approach. It treats the context window as main memory and pages information in and out from external storage, mirroring how operating systems manage virtual memory. Main context holds system instructions, message queues, scratchpads; external context holds information retrieved via function calls when something outside the active window is needed. The OS framing isn't purely metaphorical; the implementation borrows from those concepts in ways that matter at the code level.
MemoryOS, a 2025 system, formalizes a three-tier hierarchy: short-term memory for recent dialogue, mid-term for topic-based interaction history, long-term for stable user and agent characteristics. MemoryBank applies Ebbinghaus Forgetting Curve theory to dynamically adjust memory strength by time and significance, a cognitively grounded eviction policy. That borrowing from cognitive science rather than purely from systems engineering is the most methodologically interesting trend in this space right now, and it's worth watching carefully.
MemOS, published July 2025, introduces MemCubes: memory units carrying provenance and versioning metadata alongside content, enabling principled lifecycle management for what gets stored, updated, and retired. A-Mem operates at the episode level, constructing structured notes per exchange, linking them through LLM-assisted analysis into a directed graph, and expanding retrieval to neighboring nodes. Graph structure enables relational reasoning, not just semantic similarity matching. That's a qualitatively different capability, and it changes what questions you can ask of the system.
The pattern across all of it: no single storage paradigm handles all retrieval tasks equally well. Hybrid vector-graph stores keep appearing in both research and production because they stop trying to force every problem through one representation.
Production Frameworks: What Teams Actually Use to Implement Memory
Mem0 operates as a managed drop-in memory API with three-tier scopes, user, session, and agent, and a hybrid store combining vectors, graph relationships, and key-value lookups. Its retrieval layer blends semantic similarity, BM25 keyword matching, and entity matching. Benchmarks from the Mem0 team show 91% lower p95 latency and 90% token reduction compared to full-context prompting. That tracks: targeted chunks beat entire conversation histories in both cost and precision.
Zep and Graphiti are optimized for agents that reason about how facts change over time. The temporal knowledge graph is the central abstraction, not just what is true, but when it became true and whether it still is. For applications where fact currency matters, that distinction is significant and underappreciated in most product conversations.
LangChain's LangMem offers deep coupling with the LangGraph agent framework. The typology-first approach maps onto the cognitive taxonomy described earlier; procedural memory, agents rewriting their own instructions, is its most distinctive capability and the one most often overlooked by teams evaluating the framework.
Letta targets long-running agents where context management needs to be a first-class concern rather than something bolted on after the fact. Microsoft Semantic Kernel and Kernel Memory target enterprise deployments with strong Azure integration. Cognee serves local-first, privacy-critical contexts where data cannot leave the local environment.
None of these is universally better. LangMem is framework-coupled and typology-rich. Mem0 is general-purpose, optimized for retrieval pipeline extensibility. Memobase centers on user-profile-oriented, time-aware personalization. For teams working at the application layer, Vellum offers a practical interface for wiring together context windows, retrieval systems, and user-specific knowledge without building the plumbing from scratch, sitting alongside these tools as a production-oriented option for teams that want workflow-level control without assembling every component independently.
Each of these tools reflects a different prioritization. Knowing which prioritization matches your actual requirements is the decision that precedes the framework choice, and that sequencing is the part teams most often get backwards.
The Research Frontier: Architectures Rethinking Where Memory Lives
The most structurally interesting open problem is parametric memory: can a model update what it knows while it's running, rather than requiring a full training cycle?
Google's Titans architecture and the MIRAS system, both emerging in late 2025, approach this directly, allowing models to update core memory parameters during active inference. The mechanism uses a "surprise metric" to determine which information warrants permanent encoding, a computational analog to how humans preferentially encode unexpected or salient events over routine ones. The papers themselves are measured about whether that analogy holds at scale. Measured scientific tone can mean two things: genuine epistemic humility, or results that are noisier than the framing suggests. Both interpretations are worth keeping in mind as these systems move from research to production.
Dynamic context windows are a parallel direction. Rather than fixed-size windows, these systems resize active context based on task demands. A simple factual lookup and a multi-step reasoning task don't require the same memory budget; a static window either wastes capacity on the former or constrains the latter. Treating every query as equivalent is a resource allocation problem that got baked into the architecture early and is now being carefully undone.
Graph memory, represented most prominently by A-Mem and Zep, handles relational and temporal reasoning in ways flat vector retrieval cannot. Flat retrieval finds semantically similar content; graph retrieval finds structurally related content, tracking how entities connect and how those connections evolve over time. The two approaches complement each other, which is why hybrid architectures keep appearing independently across both research groups and production teams.
There are also emerging security challenges the field is only beginning to formalize: memory poisoning attacks, GDPR compliance for stored personal data, the problem of stale facts the system continues to surface after they've been superseded. For systems accumulating user data at scale, these are not theoretical concerns. The architectures being built now will either accommodate them or require expensive retrofitting later, and the cost difference between those two outcomes is substantial.
Scope, Persistence, and Precision: How the Tradeoffs Shape What an AI Can Know
Every mechanism covered here can be evaluated on three dimensions: scope, how much information can be stored; persistence, how long it survives; and precision, how reliably the right information gets retrieved when it's needed.
The context window offers narrow scope, no persistence across sessions, and high precision for whatever fits within it. It's the right instrument for in-session coherence and a poor one for almost everything else.
RAG and external vector stores offer broad scope, high persistence, and precision bounded entirely by retrieval quality. The knowledge base can be enormous and continuously updated; whether the model actually benefits depends on how well the retrieval layer performs, which loops back to every point made earlier about chunking, embedding models, and metadata.
Parametric weights cover the entire training corpus, persist indefinitely, and offer the lowest precision for specific facts. They're why the model can discuss medieval history without a database lookup, and also why it will confidently misstate a specific date. Breadth and granular reliability trade against each other in ways that aren't fully solvable at present.
Episodic stores offer moderate scope, configurable persistence, and precision that depends on embedding quality and metadata design. Graph memory scales with graph size, achieves the highest precision for relational and temporal reasoning, and costs the most to implement well.
No architecture wins on all three dimensions simultaneously. This isn't a temporary limitation waiting for a better paper; it's a structural feature of the problem space. Every real system is a deliberate set of tradeoffs, which is why layered hybrid approaches dominate production.
Before any implementation decision, the conversation worth having is this: what does this system need to remember, for whom, for how long, and how stale can that information get before it becomes a liability? Teams that skip that conversation, or have it vaguely, tend to discover its importance through the cost of fixing architecture they didn't think carefully enough about the first time.
