An agent memory architecture determines more than where conversation history is stored. It decides which observations become memories, what type they become, who may retrieve them, when they expire and whether they can influence an action without being mistaken for current truth.
A vector database can support one part of this architecture. It is not the architecture itself. Production memory needs a lifecycle around storage and retrieval.
The Four Types of Agent Memory
Cognitive terminology is useful as an engineering analogy, provided the categories are implemented as explicit contracts rather than claims that an AI remembers exactly like a person.
Working memory
Working memory is the bounded information available during the current run: instructions, the active plan, recent observations, selected evidence and a snapshot of relevant state. It answers:
What must the agent consider now?
It is assembled for a task and usually disappears or is checkpointed when the run ends. Working memory should remain small enough that important constraints are not buried in accumulated transcripts.
Episodic memory
Episodic memory records what happened in a particular interaction or run. A useful episode contains more than a chat summary:
task and context
→ actions taken
→ observations and outcome
→ evaluation or feedback
→ provenance and timestamp
It can help answer: “What happened last time we tried this?” The experience remains situated. One successful episode does not automatically establish a general rule.
Semantic memory
Semantic memory contains generalized information that has been extracted or confirmed across experience: a user preference, a domain fact, a stable association or a distilled lesson.
This layer requires particular care. A model-generated summary can be a useful memory candidate, but repeated recollection does not turn it into verified knowledge. Semantic memories need sources, confidence, scope and supersession rules.
Procedural memory
Procedural memory represents reusable ways of doing work: instructions, policies, skills, tool-selection rules, templates or strategies learned from prior outcomes. It answers:
How should this kind of task be approached?
Some procedures are fixed and version-controlled by the system. Others are model-generated lessons that require evaluation before promotion. Keeping these two origins visible prevents an improvised tactic from silently becoming operating policy.
A Memory Type Is an Operational Contract
The four labels matter because each needs different persistence and retrieval behavior.
| Type | Typical scope | Written from | Retrieved by | Main risk |
|---|---|---|---|---|
| Working | run or thread | current events and state snapshot | active coordinator | context overload |
| Episodic | user, agent or task family | completed experience | similarity, recency and outcome | copying an irrelevant precedent |
| Semantic | entity or domain | consolidation and verified updates | entity lookup plus relevance | stale or unsupported facts |
| Procedural | role, workflow or organization | approved rules and evaluated lessons | task/skill routing | unreviewed behavior becoming policy |
Without these contracts, all records become text chunks with embeddings. Retrieval then mixes preferences, old events, instructions and facts without explaining their authority.
The Agent Memory Lifecycle
A robust subsystem separates six operations.
1. Observe
Capture messages, tool results, user feedback and task outcomes as immutable observations. Not every observation deserves long-term storage.
2. Select
Apply explicit write policies. Store information only when it is likely to be useful, permitted to persist and attributable to a scope such as one user, tenant, project or agent role.
Selection can combine deterministic rules and model judgment. The model may identify a possible preference; a rule can require confirmation before it becomes a durable profile memory.
3. Encode
Create a typed record rather than saving prose alone:
memory_id, type, subject, content, source,
created_at, valid_from, valid_until, confidence,
scope, sensitivity, embedding, supersedes
The original evidence should remain addressable when a memory is a summary or inference.
4. Consolidate
Repeated episodes can be distilled into a semantic or procedural candidate. Google’s Generative Agents work used reflection to synthesize higher-level observations from accumulated experience. Architecturally, this is a promotion step, not merely compression.
Consolidation should preserve counterexamples. If four tasks succeeded with a strategy and the fifth failed, a summary that stores only “this strategy works” creates false confidence.
5. Retrieve
Retrieval should combine more than vector similarity:
eligible scope and permissions
→ memory type required by the task
→ entity and temporal filters
→ relevance, recency and importance
→ diversity and conflict handling
→ bounded context assembly
The best semantic match may belong to another tenant, have expired or describe a failed attempt. Filtering precedes ranking.
6. Revise and forget
Memories need correction, supersession, retention and deletion. A changed preference should not coexist with the old preference as two equally valid chunks. Sensitive memories may require shorter retention or must never be written at all.
Forgetting is therefore an architectural capability, not a storage failure.
Memory Is Not the Model Context
Long-term memory is a collection from which the system selects. Model context is the temporary view assembled for one call.
memory stores + current state + retrieved evidence + instructions
→ context assembler
→ bounded model context
This boundary supports model portability. The organization can reconstruct context for a different model rather than relying on a provider-specific conversation session as the only copy of prior experience.
It also supports observability: the run can record which memories were selected and whether the final action actually depended on them.
Memory Is Not Authoritative State
An agent may remember that an invoice was unpaid yesterday. The billing system may show that it was paid today. It may remember that a character possessed a key in an earlier scene, while accepted world state records that the key has since been lost.
Memory informs; state constrains.
The Stateful AI Agents architecture gives authoritative fields explicit owners and transitions. Memory can propose a state lookup or supply a precedent, but it should not overwrite current domain truth through retrieval alone.
Memory Is Not Graph RAG
Graph RAG retrieves connected evidence from a corpus. Agent memory retrieves prior experience or reusable personal and procedural information. Both may use graphs and embeddings, but their contracts differ.
A graph becomes particularly valuable when memories refer to stable entities:
(episode)-[:CONCERNED]->(customer)
(episode)-[:USED]->(procedure version)
(episode)-[:PRODUCED]->(outcome)
(semantic memory)-[:SUPPORTED_BY]->(episode)
(new memory)-[:SUPERSEDES]->(old memory)
These relations support provenance and conflict detection. They do not automatically make every extracted memory authoritative. The comparison in Graph RAG vs. Agent Memory vs. World State defines the three authority levels.
Who Controls Memory?
The model can recommend what to remember and recall, but an external coordinator should enforce persistence policies. Otherwise, the agent can write its own unverified conclusion, retrieve it later and treat the repeated text as corroboration.
A controlled memory write looks like:
model proposes memory candidate
→ coordinator checks scope, consent and sensitivity
→ validator checks schema and evidence
→ store writes typed record
→ audit event records the decision
This is the same process-sovereignty principle used throughout AI Agent Architecture : probabilistic models contribute interpretation, while the production system owns acceptance and persistence.
Agent Memory Architecture Checklist
Before adding more storage, verify that the system can answer:
- Which of the four memory types does each record represent?
- Who or what is allowed to create and retrieve it?
- Which source and episode support it?
- Is it advisory memory, retrieved evidence or authoritative state?
- How are contradictions and superseded records handled?
- When does the record expire or require reconfirmation?
- Can a user inspect, correct and delete personal memory?
- Can the runtime explain which memory influenced an action?
- Can memory be migrated independently of the model provider?
Useful agent memory is not a transcript that grows forever. It is a governed subsystem that converts selected experience into bounded, typed and revisable context—without allowing recollection to become truth by accident.