An agent needs context to continue conversations, resume processes after an interruption, and take relevant information from earlier interactions into account. But too much memory creates new problems: stale assumptions, unnecessary data retention, rising costs, and information that resurfaces outside its original context.
A customer reports a faulty delivery. The agent checks the order, identifies the affected item, and prepares a replacement shipment. Before the action is approved, the session breaks off. On the next invocation, the agent should not start over — it should know which steps have already been completed and which approval is still missing.
A few weeks later, the same customer gets in touch again. Now it can be helpful to know their preferred language or communication channel. The old delivery address, a one-off goodwill exception, or an earlier complaint, by contrast, should not flow unchecked into the new case.
Both are frequently lumped together under the term agent memory. Technically and organisationally, however, these are different jobs. The running state of a process, permanently stored preferences, confirmed business data, and audit logs each need their own rules.
An agent should not retain as much as possible. It should reliably distinguish what matters only for the current case from what may still be relevant later.
State holds the running process together
State describes the current condition of an agent execution. It ensures that a process does not have to start over after an interruption, a system error, or a human approval.
In an order process, state might record, for example:
- which order and line item have already been identified
- which tool calls have completed successfully
- which intermediate result exists
- which approval or action is still outstanding
If a connected system fails, the agent can continue from that point instead of re-executing steps that are already done.
Modern agent frameworks often store such states as checkpoints. LangGraph separates thread-scoped checkpoints from an overarching memory store. The checkpoints hold the state of one concrete run, while the store can make information available across multiple sessions.
State therefore mainly answers where the process stands, what has already been done, and with which data the next step must continue. This information does not automatically have to be retained permanently. Once the case is closed, a large part of the state can be deleted or archived after a defined recovery period.
Memory begins where context outlives sessions
Long-term memory serves a different purpose. It stores selected information that can also be helpful in later conversations or processes. This includes, for example, a confirmed language preference, recurring constraints, or explicitly saved information about a longer-running initiative.
Current memory architectures therefore often distinguish between short-term, thread-scoped context and long-term, cross-session memory. Short-term memory holds a running interaction together. Long-term memory is organised in its own storage areas and can be retrieved again in later threads.
This distinction matters, because information that is useful today does not automatically stay useful forever. That a customer asks a question about product A today does not mean this product will still be their priority in six months. That an employee communicates in English for a single project is not yet a general language preference.
Long-term memory should therefore not consist of a complete copy of all previous conversations. It is a deliberately selected and managed layer.
More context does not automatically make an agent better
One simple memory solution is to hand the model as much conversation history as possible. For short interactions, that can work. With every additional conversation, however, volume, cost, and the amount of irrelevant information grow.
Long contexts can distract models from the actual task. Stale or peripheral content competes with current information, while response times and processing costs rise. Even when the technical context window is large enough, the full history does not automatically improve the result.
For a service agent, not every word from five earlier conversations is relevant. What matters more is the current case status, confirmed customer information, and a few fitting memories. A complete history, by contrast, also contains greetings, misunderstood questions, corrected details, and information that was only needed for a closed case.
The real architectural problem is therefore not just storage. What is decisive is selection: what gets written, how is it condensed, and under what conditions is it retrieved again later?
A complete history is not yet a good memory. A production memory system must preserve what is relevant and be able to deliberately forget what is not.
Four stores with different jobs
In production agent systems, state, long-term memory, systems of record, and audit logs should not be mixed. All four store information, but for different purposes.
| Store | Typical information | Job | Sensible retention |
|---|---|---|---|
| Runtime state | Case ID, completed steps, tool results, pending approval | Continue a process or resume it after errors | Until completion plus a defined recovery period |
| Long-term memory | Confirmed preferences, recurring conditions, relevant experience | Continuity across multiple sessions | Purpose-bound, reviewable, with an expiry or review rule |
| System of record | Address, contract status, order, opportunity, entitlement | Binding business truth | Per the rules of the respective CRM, ERP, or line system |
| Audit log | Tool calls, changes, approvals, timestamps, technical results | Traceability and error analysis | Per internal operating and retention rules |
This separation prevents agent memory from quietly becoming a parallel CRM or ERP. A delivery address belongs in the leading customer system. The agent can retrieve it for a case, but should not additionally treat a possibly outdated copy as its own memory.
The same applies to audit information. A complete log is important in order to investigate a case later. But it should not automatically flow as context into every future decision.
State answers where a case stands. Memory keeps selected information available for later interactions. The system of record holds confirmed business data. The audit log documents what actually happened.
Stale memories are harder to detect than missing ones
An agent that does not know a piece of information can ask or call a source system. It becomes harder when a stored piece of information sounds plausible but is no longer valid.
A customer was interested in a particular product line in the spring. Later, the project was realigned and a different contact took over the decision. The agent finds both pieces of information, but may not recognise that the later state supersedes the earlier assumption.
A benchmark published in 2026 examined exactly this problem. The systems tested were often able to retrieve updated information, but failed to reliably recognise older assumptions as superseded and to adjust their subsequent behaviour accordingly. Even the best system examined reached only 55.2 percent across the various tests.
For the enterprise context, this is an important distinction. Retrieval and state resolution are not the same thing. The agent can see a new piece of information and still keep working on an old assumption.
A robust memory system therefore needs more than semantic search. It has to account for changes, contradictions, and temporal validity. New information must not simply be filed alongside older entries when both describe the same matter differently.
Derived assumptions must not become master data
Memory entries do not always arise from an explicit instruction. Some systems automatically extract relevant facts, preferences, or summaries from conversations and store them for later.
That is convenient, but it introduces a new quality tier. A directly confirmed statement differs from a model-side interpretation. "Please contact me by email this time" is not automatically the same as "the customer generally prefers email". "The budget is tight at the moment" should not turn into a permanent customer attribute.
Memory entries therefore need a recognisable provenance. Was the information explicitly confirmed by the user, taken from a leading system, or derived by the model from a conversation? How confident is that derivation? For what purpose was it stored, and how long should it apply?
Derived information can be useful as a working hypothesis. But it should not quietly become a binding fact or overwrite confirmed data in CRM, ERP, or other line systems.
Persistence changes the security model
Memory does not only improve continuity. It also extends the window in which manipulated or faulty information can take effect.
Microsoft therefore describes persistent agent memory not merely as a data store, but as a potential configuration layer for later behaviour. Information stored today can influence tool selection, a decision, or a response in a later session, even though the original context is long out of sight.
This also changes the impact of prompt injection and faulty content. A manipulated instruction does not have to trigger a harmful action immediately. It can first be stored as a memory and only be retrieved weeks later in a different context.
Memory should therefore be architecturally isolated by user, organisation, agent, and purpose. A subagent does not automatically need access to all memories of the parent system. Boundaries should be enforced through permissions, separate namespaces, and controlled identities — not merely through an instruction in the prompt.
The write path needs protection too. Credentials, API keys, payment information, and other particularly critical data should not end up in a general memory store in the first place. For stored entries, source, timestamp, identity, and change history matter just as much as the ability to correct and delete.
Writing and retrieval need their own rules
With classical databases, the conversation often starts with storage. With agent memory, retrieval is at least as important. Correctly stored information can still lead to a poor decision in the wrong context.
Before writing, it should be checked whether a piece of information is relevant in the long term, permissible for the intended purpose, and sufficiently confirmed. A single statement from a conversation should not automatically become a permanent memory. For uncertain derivations, it can be more sensible to ask a clarifying question or to flag the entry with lower confidence.
Retrieval needs a fresh assessment. Does the memory fit the current user, tenant, and process? Is it still current? Does it contradict a leading system or newer information? May it become visible in this context and influence an action?
Memory should be treated as possible context, not as incontestable truth. Users should also be able to view, correct, and delete stored information.
This separation between writing and retrieval prevents information, once stored, from permanently retaining the same weight. The value of a memory always depends on the current context as well.
Three decisions before implementation
Before an agent is given long-term memory, three questions should be settled:
-
What continuity does the use case need? An agent that summarises a document once probably needs no long-term memory. A service agent that accompanies multi-stage cases over several days, by contrast, needs stable state and possibly a few pieces of cross-session information.
-
Which system remains authoritative for which information? Contract status, contact details, orders, and entitlements still belong in the respective line systems. Agent memory can supplement confirmed preferences or point to this data, but should not build a second, more poorly maintained business database.
-
What does the lifecycle of a memory look like? Every memory type needs rules for creation, update, review, expiry, and deletion. It must likewise be traceable who corrects faulty entries and which memory influenced a response or action.
Only then does the choice of technical store become relevant. Vector database, key-value store, relational database, or specialised memory service solve different parts of the task. None of these technologies decides on its own what an agent should retain.
A good memory is deliberately limited
Agents without state lose the thread, repeat steps, and can hardly handle longer processes reliably. Agents with uncontrolled memory, on the other hand, carry old assumptions, sensitive information, and functionally unclear derivations into new situations.
The productive middle lies in a clear separation. Runtime state holds the current process together. Long-term memory preserves a few purpose-bound pieces of information across sessions. Leading enterprise systems remain the source for confirmed business data, while audit logs document what actually happened.
A good agent does not remember everything. It retains the right information, recognises its limits, and checks before every use whether it still holds in the current context.
The quality of agent memory therefore does not show in how much can be stored. It shows in how deliberately a system writes, updates, discards, and forgets. Only this limitation makes memory a dependable component of agentic systems.