LLM System Design · Module 7 of 17 · 7 concepts · about 10 min

Context and Memory

The model sees only what is in the prompt, so deciding what goes in, and what to keep from earlier turns, is a core design problem. This module covers the window, history, summaries and long-term memory.

At work: The part of a chat product that decides whether it remembers what you said ten minutes ago, and what that memory costs on every message.

Context window management

What it is

Many things compete for space in one prompt:

the instructions;
details about the user and their situation;
documents retrieved for this question;
the conversation so far;
results from tools;
and room for the answer itself.

Context window management is deciding, for each request, how much of each to include.

Why it matters

Every token included is paid for and read on every request, and a crowded prompt makes the model more likely to miss what matters.

The aim is the smallest context that still contains everything needed for this answer.
That is a decision made per request, not a setting chosen once.

In the real world: A research assistant's accuracy rises when the team stops including every document the user ever uploaded and sends only the three most relevant to the current question.

Filling the window because it is there. A larger window raises the ceiling; it does not change the fact that focused prompts are cheaper, faster and more accurate than stuffed ones.

Practise it: Design a ChatGPT-like Application

Token budgets

What it is

A token budget gives each part of the prompt a fixed allowance, so the total is known in advance:

a cap for instructions, for retrieved documents, for history, and so on;
a reserved amount for the answer;
a rule for what to cut first when a part goes over.

Why it matters

Without a budget, the prompt's size is whatever the inputs happen to add up to, and one long document or one long chat pushes the request over the window or the cost limit. With one, cost and latency per request are predictable, and trimming happens by a rule you chose, not by a failure at the provider.

In the real world: A support platform fixes its prompt at about four thousand tokens per request, which lets finance forecast the bill from the request count alone.

Visual example

Computed

A budget for one support request. The total, 4,100 tokens, is known before the request is built.

Token budget for one request

part of the prompt	allowance	share
Instructions (system prompt)	500	12%
Customer and order details	300	7%
Retrieved policy passages	1600	39%
Conversation history	1200	29%
The new question	100	2%
Reserved for the answer	400	10%
Total	4100	100%

Retrieved documents and history are the two parts that grow without limit unless capped.

BUDGET = {"system": 500, "profile": 300, "documents": 1600, "history": 1200}

def fit(part: str, text: str) -> str:
    """Trim one part of the prompt to its allowance."""
    tokens = encode(text)
    return decode(tokens[: BUDGET[part]]) if len(tokens) > BUDGET[part] else text

Trimming by cutting off the end. For documents, the end may hold the answer; for history, the end is the most recent and most useful part. Decide what to drop by importance: oldest turns first, lowest-ranked documents first.

Practise it: Design a ChatGPT-like Application

Sliding window over the history

What it is

The simplest way to stop a chat growing for ever is a sliding window: send only the most recent turns.

Keep the system prompt.
Keep the last N exchanges, or as many as fit the history budget.
Drop everything older.

Why it matters

It is trivial to build, and it makes the cost of each turn constant once the window is full.

Its weakness is plain: anything said before the window is gone.
If the customer gave an order number at the start and asks about 'that order' twenty turns later, the model no longer has it.

In the real world: A chatbot keeps the last ten exchanges; it handles most conversations perfectly and loses the thread only in the long ones, which is what the next technique fixes.

def recent_history(messages: list[dict], max_tokens: int) -> list[dict]:
    """The newest messages that fit, oldest first."""
    kept, used = [], 0
    for message in reversed(messages):
        size = count_tokens(message["content"])
        if used + size > max_tokens:
            break
        kept.append(message)
        used += size
    return list(reversed(kept))

Cutting in the middle of an exchange. Dropping a question while keeping its answer, or a tool call while keeping its result, leaves the model with a conversation that does not make sense. Remove whole exchanges.

Conversation summarisation

What it is

Conversation summarisation replaces the older part of a chat with a short summary written by a model:

when the history passes a threshold, summarise the oldest turns;
keep the summary plus the most recent turns in full;
update the summary as the chat goes on.

Why it matters

It keeps the gist of the whole conversation (the customer's name, the order in question, what has already been tried) at a small, fixed cost.

It combines well with a window: recent turns word for word, older ones in summary.
The price is an extra model call now and then, and the loss of exact wording.

In the real world: A telecom assistant summarises after every fifteen turns; an agent taking over a long chat reads the same summary and does not ask the customer to repeat everything.

Visual example

Computed

Total input tokens over one 40-turn chat, three ways. Sending everything costs several times more than either alternative.

System 500 tokens, question 50, answer 250. Cost at 100,000 such chats, medium tier input price.

strategy	input on turn 40	total input	cost	what is lost
Send the whole history	12,250	256,000	₹64 lakh	Nothing
Sliding window: last 6 exchanges	2,350	87,700	₹21.9 lakh	Everything older than the window
Summary + last 4 exchanges	2,050	77,800	₹19.5 lakh	Exact wording of older turns

Summary of earlier conversation (kept in place of 14 turns):
  The customer, Priya, is asking about order 88213, placed on
  3 March. It arrived 40 minutes late. She has been told the
  delivery fee will be refunded and wants to know when.

Recent turns (kept in full):
  User:      it's been three days and nothing in my wallet
  Assistant: ...

Summarising on every turn. That adds a model call to every message. Summarise only when the history crosses its budget, and do it in the background so the user is not kept waiting.

Practise it: Design a ChatGPT-like Application

Long-term memory

What it is

Long-term memory is information kept across separate conversations. It is not in the model; it is in your database:

facts about the user (prefers Hindi, is vegetarian, lives in Pune);
summaries of past conversations;
decisions and preferences stated earlier.

At the start of a request, the relevant memories are looked up and placed in the prompt.

Why it matters

It is what lets an assistant feel as if it knows you. But it is a product feature with real costs: every stored fact is personal data to protect and delete on request, a wrong or stale memory makes every later answer worse, and users must be able to see and correct what is remembered.

In the real world: A grocery assistant remembers that a customer is allergic to peanuts and flags it in every recipe suggestion, and offers a settings page listing exactly what it has remembered.

Visual example

Memory

Only the first is inside the model call. The rest are yours to store.

Three kinds of memory and where each lives

kind	lasts	stored in	example
Context window	One request	The prompt	The question being answered
Conversation history	One conversation	Your session store	The order number given earlier
Long-term memory	Across conversations	Your database	The customer prefers Hindi

Storing everything and inserting all of it. Memory is a retrieval problem: store selectively, fetch only what is relevant to this question, and prefer recent facts over old ones when they conflict.

Storing conversations

What it is

Conversations are stored in your own database, one record per message:

the conversation it belongs to and its position;
who wrote it (user, assistant, tool);
the text, the model and prompt version used, and the token counts.

Because the model keeps nothing, this store is the conversation.

Why it matters

Any server can then handle any message: it loads the history, builds the prompt and calls the model.

That is what lets the system scale by adding servers.
The store also feeds everything else: cost reports, debugging and the examples used for evaluation.

In the real world: A chat product survives a full redeployment in the middle of the day without a single user losing their conversation, because every message was already in the database.

Keeping conversation state in the memory of one server. The user's next message may reach a different server, or the first may restart, and the conversation is gone.

Message tree branching

What it is

Chat products let users edit an earlier message or regenerate an answer. Each creates a new path from that point, so a conversation is a tree, not a single line:

each message records its parent message;
an edit or a regeneration adds a sibling under the same parent;
the history sent to the model is the path from the root to the message being answered.

Why it matters

If messages are stored as a flat list, an edit either overwrites the old branch or mixes two versions of the conversation into one history. Storing the parent of each message keeps every branch intact and makes 'which history do I send?' a simple walk up the tree.

In the real world: A user edits their third message to fix a typo; the assistant answers the corrected version, and the earlier branch is still there if they switch back.

Building the history by sorting messages by time. After an edit, that interleaves messages from two different branches, and the model is shown a conversation that never happened.

Practise it: Design a ChatGPT-like Application