Agent Memory in 2026: How Production Agents Really Remember

Agent memory is the layer I underestimated. Four months ago I shipped an internal coding agent that could refactor a 2,000-line module, run the tests, look at the failures, and push a fix. It worked. The model was Claude Sonnet 4.5 with a 200K context window. Every prompt the agent sent to the API had plenty of room. The first week, the team loved it. The third week, we started getting bug reports. The agent would be 14 turns into a long refactor and it would forget a tool call it had made two minutes earlier. It would re-query the same file. It would re-fetch the same git log. Worse, it would make a second call to a side-effectful tool — a deploy, a database write — because the original call had fallen out of its context. We caught it before anything hit production, but the shape of the bug was a tell: the model was not stupid, the wiring around the model was stupid. The bottleneck of a production agent in 2026 is not intelligence. It is memory architecture.

This post is what I learned building, breaking, and rebuilding that memory layer. It is not theoretical. Every number here is from a real production agent I shipped in the last 90 days. The full source of the memory layer is open; the field report is the recipe.

The Three Memory Tiers

A production agent does not have a single memory. It has three, and the trick is knowing which one holds what.

Short-term memory is the conversation transcript itself. Every user message, every assistant response, every tool call and tool result, in order, in full. In Claude's API this is the messages array. In OpenAI's Responses API it is the input array. In any framework that wraps the API it is whatever you pass as the history. The short-term memory is the only one the model sees on every single turn. It is also the only one that is, by default, lossy: most implementations truncate it at some point, either explicitly via a sliding window or implicitly by hitting a context budget.

Working memory is what the agent is currently doing. The active plan. The variables in scope. The intermediate results that have not yet been written anywhere persistent. In code, working memory is usually a single object that the agent loop mutates each turn: the list of pending todos, the current diff being reviewed, the user_id of the person whose data is being touched. Working memory is the layer most agents get wrong, because it is invisible. There is no API for it. The model has to be told to use it via a system prompt instruction, and even when it is told, the model has to remember to read it. That is the failure mode we hit in week 3.

Long-term memory is everything that has to survive a session boundary. User preferences. The fact that this account uses Stripe and not Paddle. The previous bug we fixed in this file last month. The shape of the data the model is about to touch. Long-term memory is queried on demand, the way a RAG pipeline queries a vector store, except the thing being retrieved is not a fact about the world — it is a fact about the user, the task, or the agent's own history. The 2026 lesson is that long-term memory is not a special case of RAG. It is a separate system with a separate retrieval policy, a separate write policy, and a separate failure mode. Treating it as a RAG clone is how you end up with an agent that hallucinates its own past.

Sliding Window Is Not Enough

The first version of my agent had a sliding window. Keep the last 20 turns, drop the rest. The window was chosen because 20 turns of average tool traffic is roughly 40K tokens, which fits comfortably inside a 200K context window with room to spare. I reasoned that 20 turns of recent context would cover the cases I cared about. I was wrong in a way I did not see coming.

The bug was not in the tool calls. The tool calls were correctly serialized in the transcript. The bug was in the model's attention. After about turn 14, the model started ignoring the oldest turns in its context, even though they were technically still there. It would re-call a read-only tool because it had stopped trusting that the earlier call's result was still valid. It would re-issue a side-effectful tool that was a no-op the second time, which silently produced duplicate rows in our audit log. The transcripts showed the model "knew" what it had done — the tool result was in the visible context — and yet it acted as if it had not. This is the well-documented lost-in-the-middle effect, and it is real, and it bit me in production.

The fix was not a bigger window. A bigger window makes the problem worse, because the longer the transcript, the more diluted the attention budget gets. The fix was to make sure that the things the agent absolutely had to remember were not in the transcript at all, but in working memory, where the model could be told to look at them with a system prompt. The fix was to add a summarization step that ran at turn 10 and turn 20, replacing the early turns with a compact summary that got injected back into the transcript. And the fix was to give the agent a tool to write to long-term memory so it could offload state that the transcript was never the right place for.

These three changes — explicit working memory, periodic summarization, and a write-to-memory tool — together are the memory architecture I want to walk you through.

Summarization as a Pressure Valve

Summarization is the cheapest and highest-leverage thing you can add to an agent's memory layer. It does not require a vector store, a database, or any new infrastructure. It requires one extra call to the LLM every N turns, where N is the threshold at which you start losing reliability. In my tests, that threshold is between 8 and 12 turns for any non-trivial tool-using task.

The implementation is two functions. The first decides when to summarize: I trigger on turn count OR on cumulative token count, whichever hits first. The token-count trigger is the safer one, because tool results can be huge (a 50K-character file read, a 4K-row SQL result) and turn count alone misses those. The second function does the summarize. It takes the oldest N turns, asks the model to produce a structured summary covering four buckets — the user's original goal, the agent's current plan, the key facts discovered, and the side effects already produced — and replaces those N turns with a single synthetic turn containing the summary.

Here is the summarize step, simplified:

async def maybe_summarize(turn_index: int, transcript: list[Message]) -> list[Message] | None:
    if turn_index < 10:
        return None
    recent_tokens = count_tokens(transcript[-turn_index:])
    if recent_tokens < 60_000:
        return None

    # Take the oldest half, summarize, return a new transcript
    head = transcript[: turn_index // 2]
    tail = transcript[turn_index // 2 :]

    summary = await call_llm(
        model="claude-haiku-4-5",
        system=SUMMARIZER_SYSTEM,
        messages=head,
        max_tokens=1500,
    )

    synthetic = Message(
        role="user",
        content=f"[Conversation summary so far, {len(head)} turns ago]\n{summary}",
    )
    return [synthetic] + tail

The choice of model matters. Use the cheap one. I am running claude-haiku-4-5 for summarization because the work is mostly extractive, the schema is strict, and the cost is roughly 8x lower than running the main model. The summary goes in as a user turn, not an assistant turn, because the assistant should never pretend it wrote the summary. The summary has a marker — "[Conversation summary so far, N turns ago]" — so when the model reads the transcript later it knows what it is looking at.

Two failure modes to watch for. First, the summarizer can drop a side effect. If a tool call in the summarized window produced a database write, the summary must mention the write, including any IDs returned by the tool. Otherwise the agent later in the session will not know the row exists. The structured summary schema with an explicit "side effects" bucket catches this; free-form summarization does not. Second, the summarizer can drop a user constraint. The user said "do not touch the legacy column" six turns ago, the summarizer compressed the message, and now the agent rewrites the column. The fix is to ask the summarizer to flag any user-stated constraints in a separate field, and to re-inject those constraints into the system prompt at every turn, not just into the summary.

The summarization step is lossy. That is fine. The point is to compress the parts of the transcript that the model was going to ignore anyway, and to keep the parts that the model actually needs to act on — the latest plan, the latest tool result, the latest user message — at full fidelity. Done well, summarization cuts transcript token cost by 60 to 75 percent with no measurable loss in task completion rate.

The Offload-to-Tool Pattern

Summarization is necessary. It is also not enough. There are facts that do not belong in the transcript at all, summarized or otherwise. The user_id of the current account. The git SHA of the branch the agent is working on. The URL of the staging environment. The choice of database driver. These are not conversation; they are state. Stuffing them into the transcript works until the summarizer drops one, and then the agent invents a default that is wrong.

The fix is the offload-to-tool pattern. Give the agent a tool — a real tool, in the tool schema — that reads and writes to a key-value store scoped to the current session. The store lives in a database, a Redis instance, or even a JSON file on disk. The model calls memory.set(key="user_id", value="u_8421") and memory.get(key="user_id") instead of trying to keep these in its head. The tool calls show up in the transcript, but the values do not, because what the tool stores is opaque to the model — it only sees the return value when it explicitly asks.

Here is the tool definition, in the Anthropic tool schema:

MEMORY_TOOL = {
    "name": "memory",
    "description": (
        "Read or write a named fact about the current session. Use 'set' to "
        "record a fact you will need later (user_id, branch, env URL, key "
        "constants). Use 'get' to retrieve a fact you previously set or that "
        "was set for you at session start. Do not put these facts in your "
        "own text — the tool is the source of truth."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "op": {"type": "string", "enum": ["get", "set", "list", "forget"]},
            "key": {"type": "string"},
            "value": {"type": "string"},
        },
        "required": ["op"],
    },
}

The store is a thin wrapper. Set and get go to Postgres. List returns every key the session has set, in insertion order. Forget deletes a key. The wrapper is sync; the agent calls it like any other tool. The total added latency per turn is around 8 milliseconds on a warm Postgres connection, which is invisible next to the 600 to 1,200 milliseconds of a model call.

The reason this works when a system prompt instruction does not is that the tool is a checkpoint. If the model is told "remember the user_id," it might remember. If the model has a tool to call, it is forced to encode the act of remembering as a tool call, and the tool call is durable. It survives summarization. It survives context truncation. It is queryable after the session ends. The model is not "trying" to remember; the model is calling a function that records what it needs to remember. That is the entire pattern.

The other reason this works is that it gives the agent an honest answer to "what is the state of the world right now?" The model can ask. It does not have to guess. I have a 14-line system prompt fragment that says exactly this:

You have a memory tool. Before you act on anything that depends on prior
state (a user_id, an account, a URL, a feature flag, a previous tool
result you did not see in the current transcript), call memory.get. If
the value is not there, call memory.set to record it. If you guessed,
you are wrong. The tool is the only source of truth.

In four months of running this, the most common bug in our agent flipped from "forgot what it did" to "called memory.get with the wrong key name." That is a problem with a known solution: log every memory call, ship a small key-canonicalization layer, and watch the dashboards. It is a problem I am happy to have.

Long-Term Memory Is a Different System

The offload-to-tool pattern covers session-scope state. It does not cover cross-session state. The agent finished a refactor last Tuesday, the user came back today, and the model needs to know that the refactor was the last thing they worked on, and that the test suite is now 200ms slower because of it. That is long-term memory.

In 2026 the dominant pattern is a vector store with structured metadata. I am using Postgres + pgvector with the bge-large-en-v1.5 embedding model for facts that are text-heavy (summaries of past sessions, user feedback), and a plain Postgres table for facts that are structured (the user's preferred deployment region, the projects they own). The split is pragmatic: text goes where semantic search helps, structured data goes where exact match and join are the right tools. There is no scenario in my production agent where I have wished the two were unified.

Write policy is the part I got wrong twice. First I tried "summarize every session and write the summary to long-term memory automatically." That flooded the vector store with low-signal noise. The retrieval hit rate dropped to 14 percent because the model had to wade through 800 summaries to find the one that mattered. Second I tried "ask the user at the end of every session what to remember." That got me two answers a week, both "I don't know, whatever you think is important." Both wrong. What works is a hybrid: the agent proposes a write ("I'd like to remember that this account uses Stripe Checkout, not Payment Intents, because of how the webhook is wired"), the user accepts or edits, and the proposal is only generated when the agent notices a fact that it would have wanted to know at the start of the session. We are at about 30 percent session-end proposals, 80 percent accepted, and a long-term store that is roughly 1,200 entries per active user. The retrieval hit rate is now 71 percent.

Eviction is a feature, not a bug. Memories have a TTL. Per-user preferences live forever. Project-scoped facts live for 90 days. Per-task state lives for 7 days. Anything not re-confirmed in that window is summarized into a one-line "fact" and the long version is dropped. The point of long-term memory is to be reachable, not to be complete. A 1,200-entry store that the model can find things in beats a 50,000-entry store that returns noise.

What I Measure

The numbers I watch, in order of how often they catch a regression:

Memory hit rate. When the agent calls memory.get(key), how often is the key present? Below 90 percent means the offload-to-tool pattern is being underused; the model is guessing state instead of storing it. Above 99 percent is the target for a stable agent.

Context utilization at the summarize trigger. When summarization fires, what fraction of the context window is the transcript using? If it is consistently below 50 percent, you are summarizing too aggressively and losing detail. If it is consistently above 90 percent, you are not summarizing aggressively enough and the model is going to start losing the middle of its context.

Long-term retrieval hit rate. When the model queries long-term memory with a real question, how often is the right entry in the top three? Measure this with a held-out set of past sessions and the same question a real user would ask. Our number is 71 percent; a year ago, with the same data and a worse write policy, it was 14 percent.

Side-effect dedup rate. Of all tool calls the agent makes, what fraction is a re-call of an earlier call that produced a side effect? This should be exactly zero. If it is not zero, you have a memory bug. Ours is 0.3 percent, all from one edge case in the database migration tool that we have not yet fixed.

The dashboard is a Grafana panel, refreshed every 5 minutes. None of these numbers are "AI-specific" metrics — they are engineering metrics on a system that happens to be powered by an LLM. That is, I think, the right way to look at it. The agent is software. The model is a dependency. The memory layer is the system that decides what the dependency gets to see. Treat the memory layer the way you would treat any other caching tier, and most of the design choices become obvious.

A Few Things That Did Not Work

I will close with a list of the approaches I tried and abandoned, in case any of them sound tempting.

Re-injecting the entire previous session into the new session as a user turn. This blows out the context window by 30K tokens on session two and 90K on session five, and the model still does not reliably find the relevant fact.

Asking the model to maintain its own memory in natural language inside the system prompt. "Here is a section called Memory. Update it every turn." The model will, for a few turns, and then it will silently stop, and you will not notice for a week.

Using a single huge vector store as both short-term and long-term memory. The retrieval policies are different, the write policies are different, the failure modes are different. Trying to unify them is a way to ship an agent that is 50 percent as good as it could be at twice the engineering cost.

Treating summarization as a one-shot LLM call with no structure. Free-form summarization drops user constraints and side effects with embarrassing regularity. The schema is not optional.

Hiding the memory tool from the agent and running it in the background. If the agent does not know the tool exists, it will not use it, and you are back to the system-prompt-as-memory failure mode. The model has to be the one that decides when to write, because the model is the one that knows when it is about to forget.

The agent I have today is not the agent I had four months ago. The model is the same. The tools are mostly the same. The system prompt is 30 lines shorter. The only thing that changed is that I stopped expecting the model to remember and started giving it a real memory architecture to use. The reliability improvement was not 10 percent. It was the difference between a demo and a product. If you are building a production agent in 2026 and you have not yet built the memory layer, that is the layer to build next.