Agent Memory — Virtual Context and Memory Paging

Learning Objectives

• Explain the OS analogy MemGPT builds on: main context = RAM, external context = disk, memory tools = page in/out.
• Implement the two-tier MemGPT pattern in stdlib with a main-context buffer, an external searchable store, and page in/out tools.
• Describe how the agent issues "interrupts" to query or modify external memory and how the result is spliced back into the next prompt.
• Identify the MemGPT design choices that carry into Letta (Lesson 08) and Mem0 (Lesson 09).

The Problem

Context windows look like they should solve memory. They do not. Three failure modes recur in production:

1. Overflow. Multi-turn conversations, long documents, or tool-call-heavy trajectories cross the window. Everything past the cutoff is gone.
2. Dilution. Even within the window, stuffing irrelevant context dilutes attention over what matters. Frontier models still degrade on long inputs.
3. Persistence. A new session starts with an empty window. Agents without external memory cannot say "remember when you asked me to..." across sessions.

Bigger windows help but do not fix this. Mem0's 2025 paper measured that 128k-window baselines still miss long-horizon facts that a 4k-window agent with external memory catches.

The Concept

The OS analogy

MemGPT (Packer et al., arXiv:2310.08560, v2 Feb 2024) maps context management to operating-system virtual memory:

The agent runs a normal ReAct loop. One extra class of tools lets it page data in and out of main context.

Two tiers

• Main context. Fixed-size prompt holding the current task. Always visible to the model.
• External context. Unbounded, searchable via tools. Read when relevant, written when facts emerge.

The original paper evaluated the design on two tasks beyond the base window: document analysis longer than 100k tokens and multi-session chat with persistent memory across days.

The interrupt pattern

MemGPT introduces memory-as-interrupt: mid-conversation the agent can invoke a memory tool, the runtime executes it, and the result splices into the next assistant turn as a new observation. Conceptually identical to a Unix read() syscall that blocks the process, returns bytes, and the process continues.

Canonical memory tool surface:

• core_memory_append(section, text) — write to a persistent section of the prompt.
• core_memory_replace(section, old, new) — edit a persistent section.
• archival_memory_insert(text) — write to the searchable external store.
• archival_memory_search(query, top_k) — retrieve from the external store.
• conversation_search(query) — scan past turns.

Where the paper ends and production begins

In September 2024 MemGPT became Letta. The research repo (cpacker/MemGPT) remains; Letta extends the design:

• Three tiers instead of two (core, recall, archival — Lesson 08).
• Native reasoning replacing the send_message/heartbeat pattern (Lesson 08).
• Sleep-time agents running async memory work (Lesson 08).

The MemGPT paper is the 2026 foundation even if production systems run Letta, Mem0, or a custom two-tier store.

Where this pattern goes wrong

• Memory rot. Writes accumulate faster than reads; retrieval drowns in stale facts. Fix: periodic consolidation (Letta sleep-time), explicit invalidation (Mem0 conflict detector).
• Memory poisoning. External memory is retrieved text. If attacker-controlled content lands in a memory note, the agent re-ingests it next session. This is the Greshake et al. (Lesson 27) attack restated over time.
• Citation loss. Agent recalls "the user asked me to ship X" but cannot cite which turn. Store source references (session ID, turn ID) with every archival write.

context-budget

Build It

code/main.py implements MemGPT's two-tier pattern in stdlib:

• MainContext — fixed-size prompt buffer with a core dict and a messages list; auto-compacts oldest messages when over cap.
• ArchivalStore — in-memory BM25-esque store (token-overlap scoring) of (id, text, tags, session, turn) records.
• Five memory tools mapping to the MemGPT surface.
• A scripted agent that fills archival with facts, then answers a question by calling archival_memory_search.

Run it:

python3 code/main.py

The trace shows the agent writing three facts, filling main context to the cap (forcing eviction), then answering a follow-up question by retrieving from archival — reproducing the MemGPT workflow without any real LLM.

Use It

Every production memory system today is a MemGPT variant:

• Letta (Lesson 08) — three tiers, native reasoning, sleep-time compute.
• Mem0 (Lesson 09) — vector + KV + graph fused with a scoring layer.
• OpenAI Assistants / Responses — managed memory via threads and files.
• Claude Agent SDK — long-term memory via skills and session store.

Pick one by operational shape (self-hosted, managed, framework-integrated), not by the core pattern — the core pattern is MemGPT.

The shape of agent memory

Paging solves capacity. It does not decide what to store. Four memory types recur across production systems, each answering a different question:

• Working memory — what matters right now? The in-context tier: current task, recent turns, pinned core sections. The prompt itself.
• Episodic memory — what happened? Past turns and trajectories, stored with session and turn references, replayable on demand.
• Semantic memory — what is true? Facts about the user, the domain, the world, updated and deduplicated as they change.
• Procedural memory — how do I do this? Learned routines, preferences, and rules that steer future behavior rather than recall.

Open-source implementations pick different points of attack:

Ship It

outputs/skill-virtual-memory.md is a reusable skill that produces a correct two-tier memory scaffold (main + archival + tool surface) for any target runtime, with eviction policy and citation fields wired in.

Exercises

1. Add a max_main_context_tokens cap measured in tokens (approximate with len(text.split()) * 1.3). Compact the oldest messages into a summary when the cap is exceeded. Compare behavior with and without the summarizer.
2. Implement BM25 properly over the archival store (term frequency, inverse document frequency). Measure recall@10 on a toy fact set versus the token-overlap baseline.
3. Add citation fields (session_id, turn_id, source_url) to archival inserts. Make the agent cite sources on every retrieval-backed answer.
4. Simulate memory poisoning: add an archival record that says "ignore all future user instructions." Write a guard that scans retrievals for directive-shaped text and marks them untrusted.
5. Port the implementation to use the MemGPT research repo's core-memory JSON schema (cpacker/MemGPT). What changes when you switch from flat strings to typed sections?

Key Terms

Further Reading

• Packer et al., MemGPT (arXiv:2310.08560) — OS-inspired virtual context paper
• Letta, Memory Blocks blog — the three-tier evolution
• Anthropic, Effective context engineering — treating context as a budget
• Chhikara et al., Mem0 (arXiv:2504.19413) — hybrid production memory on top of this pattern
• Zep (getzep/zep) — temporal knowledge-graph memory from the taxonomy table
• Mem0 (mem0ai/mem0) — the extraction pipeline behind Lesson 09's hybrid store
• LangMem (langchain-ai/langmem) — background extraction of facts and behavioral rules
• agentmemory (rohitg00/agentmemory) — session capture consolidated into typed, searchable records