Agentic AI · Module 5 of 41 · 12 concepts · about 20 min

Agent Memory

The context window and working memory, the kinds of long-term memory, the stores used to build them, how memories are written, scored, compressed, updated and expired, and how memory differs from RAG.

At work: What makes an assistant feel like it knows you, and the source of its most embarrassing mistakes when a remembered fact is wrong, stale or someone else's.

The context window and working memory

What it is

Two terms that describe what a model can 'hold in mind':

A token is a piece of a word. Models measure text in tokens; a page is roughly 500.
The context window is the maximum number of tokens a model can read in one call. Everything it knows about the task must fit inside it.
Working memory is what you actually put in the window for this turn: instructions, the task, recent steps, and anything fetched from storage.

Why it matters

The window is the model's entire awareness, and it is finite.

Anything not in the window does not exist for the model on that turn.
A full window is not the goal. Models attend less well to the middle of very long inputs, and every token is paid for on every turn.
So memory is really a selection problem: of everything that could be shown, what should be?

In the real world: An agent with a huge window was given a customer's full two-year history on every turn; it began quoting a cancelled address from 2024 until the history was replaced with a short profile.

Assuming a large window removes the need for memory design. A million-token window filled with every past message is slow, costly, and buries the one fact that matters.

Short-term and long-term memory

What it is

Memory is split by how long it lasts:

Short-term memory belongs to one task or conversation: the messages and tool results so far. It lives in the context window and in the saved state of the run. It ends with the task.
Long-term memory outlives the task. It is written to a store outside the model and read back in later sessions.

Why it matters

They solve different problems and fail differently.

Short-term memory keeps a task coherent across steps. Its problem is growth: it gets too long.
Long-term memory gives continuity across days. Its problems are relevance, staleness and privacy.
A model has neither by nature. Both are things your code builds.

In the real world: A tutoring app keeps the current lesson in short-term memory and only three things long term: the student's level, the topics they struggle with, and their preferred examples.

Visual example

Memory

One is the task's own record. The other is what is kept afterwards.

Short-term versus long-term memory

	short-term	long-term
Lasts for	One task or conversation	Weeks, months, indefinitely
Lives in	The context window and run state	A database outside the model
Holds	Messages and tool results	Selected facts, events and procedures
Reaches the model by	Being resent every turn	Being retrieved when relevant
Main problem	Grows too long	Wrong, stale or irrelevant recall

Saving the whole conversation as long-term memory. Most of a conversation is not worth keeping. Long-term memory should hold the few facts that will matter again.

Episodic, semantic and procedural memory

What it is

Long-term memory comes in three kinds, named after human memory:

Episodic: what happened. 'On 3 March the customer's order was late and we refunded ₹25.'
Semantic: what is true. 'The customer is vegetarian and lives in Pune.'
Procedural: how to do things. 'For damaged items, ask for a photo first.' Often stored as instructions, examples or reusable skills.

Why it matters

Each kind is stored and used differently.

Episodes are many and dated; they are searched, and old ones are summarised or dropped.
Semantic facts are few and must stay correct; a new fact replaces an old one.
Procedures change the agent's behaviour, so they deserve review before they are saved.

In the real world: A coding agent stores 'the test command for this repository is make check' as a procedure; without it, every new session spent three steps rediscovering how to run tests.

Visual example

Memory types

What happened, what is true, and how to do it.

Three kinds of long-term memory

type	holds	example	typical store	update rule
Episodic	Events	Refunded ₹25 on 3 March	Vector store with dates	Append; summarise when old
Semantic	Facts	Vegetarian; lives in Pune	Key-value or profile table	Overwrite when it changes
Procedural	Methods	Ask for a photo before a damage refund	Prompt, rules file, skills	Edit, with review

Storing everything as episodes. An agent that must re-derive 'this customer is vegetarian' from forty past orders will sometimes get it wrong. Promote repeated observations into one semantic fact.

Practise it: Design an AI Coding Assistant

Where memories are stored

What it is

Long-term memory is ordinary data in ordinary stores. The common choices:

Conversation history: the raw messages, in a table. Complete, long and rarely reread whole.
Summaries: a short written account of a past conversation.
Vector database: stores text with its embedding (a list of numbers that captures meaning), so memories can be found by similar meaning, not exact words.
Key-value store: one value per named key, such as diet = vegetarian. Exact and fast.
Relational database: tables with columns, for facts you query precisely: orders, preferences, consents.
Knowledge graph: facts as linked things: 'Asha → manages → Ravi'. Good for questions about relationships.
User profile: a short structured document about one user, loaded at the start of every session.

Why it matters

The store decides what kind of question memory can answer.

'What did we discuss about refunds?' needs search by meaning: a vector store.
'What is the customer's diet?' needs an exact look-up: a key-value store or profile.
'Who else reports to Ravi's manager?' needs relationships: a graph or SQL joins.
Most real systems use a profile for the few facts that always matter and a vector store for everything else.

In the real world: A health assistant moved allergies from a vector store to a profile field loaded on every turn, after a similarity search once failed to return the allergy at all.

Visual example

Stores

Choose by the question you need to answer, not by the technology.

Which store suits which memory

store	finds by	best for	weak at
Conversation log	Time, id	Audit, replay	Being reread by the model
Summaries	Conversation id	Continuity across sessions	Exact detail
Vector database	Similar meaning	Past episodes, notes	Exact facts, recency
Key-value store	Exact key	Preferences, settings	Open-ended recall
Relational database	Query	Structured facts with history	Free text
Knowledge graph	Relationships	Who and what is connected	Cost to build and maintain
User profile	User id	The few facts that always apply	Anything large

Putting everything in a vector database because it is the fashionable store. Similarity search is the wrong tool for exact facts: it may return 'vegetarian' or a three-year-old 'loves chicken' with nearly equal confidence.

Writing memories: what to keep

What it is

Memory writing is deciding what to save. Three common triggers:

The agent decides: it has a remember tool and calls it when it learns something lasting.
After the conversation: a separate, cheaper model call reads the transcript and extracts facts worth keeping.
Explicit request: the user says 'remember that I…'.

Whatever the trigger, save a short, self-contained statement with its date, source and the user it belongs to.

Why it matters

What you save decides what you can ever recall.

Save too little and the agent asks the same questions every week.
Save too much and retrieval drowns in trivia.
A useful test: will this still be true and useful in a month? 'Allergic to peanuts' passes. 'Is in a hurry today' does not.

In the real world: A sales assistant's extractor saved 'customer was annoyed'; three months later the agent opened a call with an apology nobody understood. The rule became: save facts and decisions, not moods.

from datetime import date

def remember(user_id: str, fact: str, kind: str, importance: float, source: str) -> None:
    """Save one self-contained fact. Called by the agent, or by a post-chat extractor."""
    memory_store.add(
        user_id=user_id,                  # every memory belongs to exactly one user
        text=fact,                        # "Allergic to peanuts", not "they said that"
        kind=kind,                        # "semantic" | "episodic" | "procedural"
        importance=importance,            # 0..1, set by the extractor
        created=date.today().isoformat(),
        source=source,                    # which conversation it came from
        embedding=embed(fact),
    )

Saving a memory that only makes sense inside its conversation: 'they prefer the second option'. Rewrite it to stand alone before saving: 'Prefers delivery after 6 pm'.

Memory retrieval and relevance scoring

What it is

Memory retrieval is choosing which stored memories to show the model this turn. A common score combines three things:

Similarity: how close in meaning the memory is to the current request.
Recency: how new it is. A memory's weight halves every so many days.
Importance: how much it matters, set when it was saved.

The top few by score are placed in the context.

Why it matters

Similarity alone picks what sounds related, not what matters.

A recent trivial memory can outrank an old vital one.
Importance lets a safety fact surface even when the request barely mentions it.
Recency stops stale events crowding out current ones.

In the real world: A grocery assistant asked for 'a snack' ranks the peanut allergy first, ahead of last week's samosa order, only because importance is part of the score.

Visual example

Scoring

score = 0.5 × similarity + 0.2 × recency + 0.3 × importance. Recency halves every 30 days.

Ranking four memories for “suggest a snack for my order”

memory	similarity	age (days)	recency	importance	score
Allergic to peanuts	0.62	90	0.13	1	0.64
Ordered samosas last Friday	0.71	6	0.87	0.2	0.59
Prefers evening delivery	0.2	3	0.93	0.5	0.44
Complained about a late order	0.15	45	0.35	0.4	0.27

With similarity alone, the samosa order would rank first and the allergy second.

def score(memory, query_embedding, today) -> float:
    similarity = cosine(memory.embedding, query_embedding)          # 0..1
    recency = 0.5 ** (days_between(memory.created, today) / 30)     # halves every 30 days
    return 0.5 * similarity + 0.2 * recency + 0.3 * memory.importance

def recall(user_id, query, k=3):
    candidates = memory_store.search(user_id=user_id, query=query, limit=20)
    ranked = sorted(candidates, key=lambda m: score(m, embed(query), today()), reverse=True)
    return ranked[:k]

Retrieving by similarity only and trusting the order. For facts that must never be missed, such as allergies or legal holds, do not rely on retrieval at all: load them on every turn from the profile.

Practise it: Design a Tool-Calling Agent Over 500 APIs

Memory compression and summarisation

What it is

Memory compression shrinks what is kept so it still fits. Three techniques:

Rolling summary: when the conversation passes a length, replace the oldest part with a short summary and keep the recent turns in full.
Consolidation: merge many episodes into one fact. Twelve vegetarian orders become 'vegetarian'.
Hierarchical summaries: summaries of days roll up into summaries of weeks.

Why it matters

Without compression, memory grows until it is too costly to read and too noisy to use.

A summary keeps the decisions and drops the chatter.
Consolidation turns evidence into knowledge, which is cheaper to retrieve and harder to miss.
Every summary loses detail, so the raw log is kept elsewhere for audit.

In the real world: A long-running coding agent's compaction dropped a file path; the agent then 'fixed' a different file. The summary prompt now lists paths, commands and error messages as never-drop items.

def compact(messages: list[dict], keep_recent: int = 10, limit_tokens: int = 20_000):
    """Summarise the oldest turns once the history is too long."""
    if count_tokens(messages) < limit_tokens:
        return messages
    old, recent = messages[:-keep_recent], messages[-keep_recent:]
    summary = llm_small(
        "Summarise this conversation for an agent that will continue it. Keep: the "
        "user's goal, facts established, decisions made, actions taken with ids, "
        "and what is still open. Drop greetings and repetition.\n\n" + render(old)
    )
    return [{"role": "user", "content": f"Summary of earlier conversation:\n{summary}"}, *recent]

Summarising away identifiers. A summary that says 'a refund was issued' has lost the order number and amount the agent needs next. Tell the summariser to keep ids, amounts and dates verbatim.

Updating, conflicts, deduplication and expiry

What it is

Stored memories need upkeep. Four problems and their fixes:

Updating: a fact changes. Overwrite it, and keep the old value with an end date if history matters.
Conflicts: two memories disagree ('vegetarian', 'ordered chicken'). Prefer the newer or the more explicit; if unsure, ask the user.
Deduplication: the same fact saved many times. Before saving, search for a near-identical memory and merge.
Expiry: some memories should end. Give each a time to live (TTL) or a review date; delete or demote when it passes.

Why it matters

A memory store that only ever grows becomes wrong.

People move house, change diet, cancel memberships.
Duplicates crowd retrieval so the top five results are one fact five times.
A confident answer built on a stale memory is worse than no memory, because nothing signals the error.

In the real world: A delivery app kept sending groceries to a customer's old flat because 'home address' was appended, never replaced; switching addresses to a keyed fact fixed it.

Visual example

Upkeep

Every stage after the first is what keeps the store correct.

The life of one memory

ExtractLasting facts from the chat
DeduplicateMerge near-identical memories
Resolve conflictNewer and explicit wins
StoreUser, date, source, expiry
ExpireDeleted or summarised

def save_fact(user_id: str, key: str, value: str) -> None:
    """Semantic facts are keyed, so a new value replaces the old one."""
    current = profile.get(user_id, key)
    if current and current.value == value:
        profile.touch(user_id, key)                       # duplicate: just refresh the date
        return
    if current:
        profile.archive(user_id, key, ended=today())      # keep history, mark it ended
    profile.set(user_id, key, value, since=today())

def save_episode(user_id: str, text: str) -> None:
    twin = memory_store.search(user_id=user_id, query=text, limit=1)
    if twin and cosine(twin[0].embedding, embed(text)) > 0.92:
        return                                            # near-duplicate: do not store again
    memory_store.add(user_id=user_id, text=text, expires=in_days(180))

Resolving a conflict silently in the agent's favour. If the profile says vegetarian and today's basket has chicken, the right move is to ask, not to guess which is true.

Privacy and personalisation

What it is

Personalisation is using what you know about a user to serve them better. It carries duties:

Isolation: a memory is readable only for the user it belongs to. The user id is a filter on every query, set by your code.
Consent and visibility: users can see what is remembered and switch it off.
Deletion: 'forget that' must really delete, in every store and index.
Minimisation: do not keep sensitive data (health, payment, identity numbers) unless the task needs it.

Why it matters

Memory is where an agent's worst data leaks come from.

A missing user filter shows one person another's history.
Laws such as the GDPR and India's DPDP Act give people the right to have their data erased.
Users who discover the agent remembered something they did not expect stop trusting it.

In the real world: A shared family tablet led an assistant to suggest one member's medication to another; the product added per-profile memory and a 'what do you remember about me?' screen.

Letting the model choose which user's memories to search. The user id must come from the authenticated session and be applied by the storage layer, where the model cannot change it.

Memory versus RAG

What it is

Both fetch text and put it in the context, which is why they are confused. They differ in what is fetched and who wrote it:

RAG retrieves from a shared body of knowledge written by people beforehand: policies, manuals, product pages. The same for every user. The agent reads it; it does not write it.
Memory retrieves from what the agent itself recorded while working: facts about this user, past events, lessons learned. Different for every user. The agent reads and writes it.

Why it matters

They use similar machinery for different jobs, so they need different care.

RAG's problems are document quality, chunking and search accuracy.
Memory's problems are deciding what to write, keeping it current, and keeping users' data apart.
A good agent uses both: RAG for 'what is the refund policy?', memory for 'what did this customer ask last time?'.

In the real world: A support agent answers 'can I return this?' from the policy via RAG, then adds 'you returned the same item in May' from memory; the two come from separate stores with separate access rules.

Visual example

Comparison

Same retrieval machinery. Different contents, owner and risks.

Memory versus RAG

	RAG	memory
Contains	Documents written in advance	What the agent learned while working
Written by	People, through an ingestion pipeline	The agent, during or after tasks
Scope	Shared by all users	Private to one user or task
Changes	When documents are republished	Continuously
Agent can write to it	No	Yes
Typical question	What is the policy?	What happened last time?
Main risk	Wrong or missing passage	Stale, conflicting or leaked facts

Storing user memories in the same index as company documents with no user filter. One search then returns a stranger's history beside the policy.

Interview questions: agent memory

What it is

The questions interviewers ask about agent memory, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: A system design round asks for 'an assistant that remembers users'; the answers that score well separate a profile from an episode store and mention deletion without being asked.

Q: Does an LLM have memory?

No. Each call sees only the text sent in that call. Short-term memory is the history your code resends; long-term memory is a store your code reads and writes. Memory is a retrieval and selection problem.

Q: What types of memory are there?

Short-term or working memory inside the context window; long-term memory outside it, which divides into episodic (events), semantic (facts) and procedural (how to do things).

Q: How would you implement long-term memory?

A structured profile for the few facts that always apply, loaded every session, plus a vector store of episodes filtered by user id and ranked by similarity, recency and importance. A post-conversation extractor writes memories; dedup, conflict rules and expiry keep them correct.

Q: How does memory differ from RAG?

RAG reads shared documents written in advance. Memory reads and writes what the agent itself learned, per user. Similar retrieval, different content, ownership, update pattern and privacy rules.

Q: How do you stop the context filling up?

Trim tool results, summarise old turns while keeping recent ones verbatim, keep ids and amounts in the summary, and fetch details on demand instead of holding them.

Q: Two memories conflict. What do you do?

Prefer the newer and the more explicit statement, keep the old one marked ended, and if the conflict matters to the action, ask the user rather than guess.

Q: What are the privacy risks?

Cross-user leakage from a missing filter, keeping sensitive data that was never needed, and failing to delete on request. Enforce the user id in the storage layer, show users what is stored, and delete from every index.

Q: How do you evaluate memory?

Test recall of planted facts across sessions, check that updated facts replace old ones, measure how often retrieved memories were actually used, and include a test that one user's fact never appears for another.

Answering 'we store the chat history in a vector database' and stopping. Interviewers then ask about a changed address or a deletion request, and that design has no answer.

Exercise: add long-term memory to a chat loop

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A product team builds this in an afternoon to decide what their assistant should remember; the forget command turns out to be the feature legal asks about first.

Your task. Give a simple chat agent memory that survives between sessions, with scoring, deduplication and a forget command.

Steps

Start with a loop that sends the conversation to a model and prints the reply.
Add a JSON file per user holding a profile dictionary and a list of episodes (text, date, importance).
After each conversation, make one model call that extracts lasting facts as structured output and saves them: keyed facts into the profile, events into episodes.
At the start of each turn, put the profile in the system prompt and add the top three episodes ranked by the similarity, recency and importance formula above. Simple word overlap is fine for similarity.
Skip saving an episode that nearly matches an existing one. Make a changed profile value replace the old one.
Add a /forget <text> command that deletes matching memories.

You are done when

Tell it your diet, restart the program, ask for a recipe: the diet is respected.
Change your diet: the old value no longer appears in the profile.
Say the same fact three times: it is stored once.
After /forget, a restart shows no trace of the fact.
Running as a second user shows none of the first user's memories.

Stretch. Replace word overlap with real embeddings and compare which memories are retrieved for five queries.