LLM System Design · Module 10 of 17 · 8 concepts · about 11 min

Caching

The cheapest and fastest model call is the one you do not make. This module covers the kinds of cache an LLM system uses, what each saves, and how each can go wrong.

At work: The first optimisation most teams reach for once a feature is live: real traffic repeats itself far more than anyone expects.

What can be cached

What it is

An LLM system has four different caches, at four different points:

Response cache: the whole answer to a request seen before.
Semantic cache: the answer to a request similar to one seen before.
Prompt cache: the provider's saved work for the unchanging start of a prompt.
Embedding and retrieval caches: vectors and search results that would otherwise be recomputed.

Why it matters

They save different things.

A response cache skips the model call entirely: no cost, no wait.
A prompt cache still makes the call, but charges less and starts faster.
Knowing which is which matters, because the first two can return a wrong answer and the last two cannot.

In the real world: A support platform uses all four: exact matches for common questions, the provider's prompt cache for its long instructions, and saved embeddings for its help-centre articles.

Visual example

Caches

The first two can serve a wrong answer; the last two cannot.

Four caches and what each one skips

cache	matches on	skips	main risk
Response (exact)	An identical request	The whole model call	Stale answers; low hit rate
Semantic	A similar request	The whole model call	Answering a different question
Prompt (provider)	An identical start of the prompt	Re-reading that part	None for correctness
Embedding / retrieval	Identical text or query	Re-embedding, re-searching	Stale after the source changes

Saying 'I would add caching' without saying which kind. They have different savings and different risks, and the interviewer's next question will be about the risk of whichever one you meant.

Response caching

What it is

Response caching stores the answer to a request and returns it when exactly the same request arrives again:

build a key from everything that determines the answer;
look the key up before calling the model;
on a hit, return the stored answer; on a miss, call the model and store the result.

Why it matters

A hit costs nothing and returns in a few milliseconds.

It works well where requests repeat exactly: classification of the same texts, common fixed questions, repeated batch jobs, evaluation runs.
It works poorly on free-form chat, where no two messages are quite identical.

In the real world: A nightly job that re-classifies a product catalogue calls the model only for items that changed; the rest are served from the cache, cutting the run from hours to minutes.

import hashlib, json

def cache_key(model: str, prompt_version: str, messages: list, settings: dict) -> str:
    payload = json.dumps([model, prompt_version, messages, settings], sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

def cached_chat(model, prompt_version, messages, settings):
    key = cache_key(model, prompt_version, messages, settings)
    if (hit := cache.get(key)) is not None:
        return hit
    answer = gateway.chat(model=model, messages=messages, **settings)
    cache.set(key, answer, ttl_seconds=3600)
    return answer

Leaving something out of the key. If the model or the prompt version is not part of it, a release keeps serving answers written by the old version. If user-specific context is not part of it, one customer can be shown another's answer.

Practise it: Design an LLM Gateway

Semantic caching

What it is

A semantic cache matches on meaning, not exact text:

embed the incoming question into a vector;
search stored questions for the closest one;
if it is similar enough (above a threshold), return that question's stored answer.

Why it matters

People ask the same thing in many ways: 'where is my refund', 'refund status?', 'when do I get my money back'.

An exact cache misses all but one.
A semantic cache catches them, which raises the hit rate a great deal on chat-style traffic.

In the real world: A bank's assistant uses a semantic cache for general questions such as branch hours, and bypasses it for anything that mentions an account, where a wrong hit would be serious.

A threshold that is too loose. 'Cancel my order' and 'cancel my membership' are close in meaning and need entirely different answers. A semantic cache will, sooner or later, give someone the answer to a different question. Use a strict threshold, apply it only to general questions that do not depend on the user, and measure wrong hits.

Practise it: Design an LLM Gateway

Prompt caching

What it is

Prompt caching is a provider feature. When many requests begin with the same long text, the provider keeps its processed form of that beginning:

the first request pays the normal price, sometimes a little more, to store it;
later requests that start with the identical text are charged a fraction of the input price for that part;
the model also starts answering sooner, because it does not re-read it.

The match must be exact from the very first token; the cache expires after minutes of disuse.

Why it matters

Long, unchanging prefixes are everywhere: system instructions, tool definitions, few-shot examples, a document being discussed across many turns. Prompt caching cuts the cost and delay of that part without any risk to correctness, because the model still produces a fresh answer.

In the real world: A team sees no saving from prompt caching until someone notices 'Today's date is...' at the top of the system prompt; moving it to the end makes most of the prompt reusable.

Visual example

Computed

One request with a 2,500-token unchanging prefix out of 3,000 input tokens, cached input charged at 10% of the normal price.

Cost per request on the medium tier (300 output tokens). The discount is illustrative.

	input ₹	output ₹	total ₹	per million requests
Without prompt caching	0.75	0.375	1.125	₹11.3 lakh
With prompt caching	0.188	0.375	0.563	₹5.6 lakh

The input side

plain: 3,000 tokens at full price = ₹0.75

cached: 2,500 at 10% + 500 at full price = ₹0.188

saving: 75% off the input cost; the output cost is unchanged

Order the prompt from least-changing to most-changing:

  1. System instructions        same for every request    } cached
  2. Tool definitions           same for every request    } prefix
  3. Few-shot examples          same for every request    }
  ---------------------------------------------------------
  4. Retrieved documents        differs per question
  5. Conversation history       differs per conversation
  6. The new message            differs every time

Putting something that changes near the top. A timestamp, a user's name or a request ID in the first line makes every prompt different from its first token, and nothing is ever reused. Everything that varies goes after everything that does not.

Practise it: Design a ChatGPT-like Application, Design an LLM Gateway

What goes in a cache key

What it is

The key must contain everything that could change the right answer:

the model and its version;
the prompt version;
the full input: messages and any retrieved context;
the settings that affect output, such as temperature;
whatever makes the answer specific to a user: their permissions, language or tenant.

Why it matters

A cache is correct only if two requests with the same key deserve the same answer.

Anything missing from the key becomes a way to serve the wrong one.
The dangerous omission is access: if two users may see different documents, an answer built from one user's documents must never be returned to the other.

In the real world: A company's internal assistant includes the user's department in the cache key after a review finds that an HR answer could otherwise have been served to an engineer who asked the same question.

Caching answers that were built from permission-filtered data under a key that ignores the permissions. It passes every test with a single test user and leaks on the first day of real use.

Practise it: Design an LLM Inference Serving System

Invalidation and freshness

What it is

A cached answer is a snapshot. It goes stale when what it was built from changes. Three ways to keep it fresh:

Time to live (TTL): every entry expires after a set time.
Versioned keys: the key includes a version of the source, so a change makes old entries unreachable.
Explicit invalidation: when a document changes, delete the entries built from it.

Why it matters

The right lifetime depends on the content.

Opening hours can be cached for a day. An order's status cannot be cached at all.
A refund policy can be cached until the policy document changes, which is what versioned keys handle neatly: update the document, bump its version, and every old answer is skipped.

In the real world: An airline caches answers about baggage rules with the rule document's version in the key; when the rules change, the next question gets a fresh answer at once.

One long TTL for everything. The first policy change then leaves the assistant quoting the old policy until entries expire, with a citation that looks perfectly current.

What a response cache saves

What it is

The saving is the hit rate times the cost of a call:

each hit avoids one full model call;
each miss costs the same as before, plus a cheap lookup;
so the average cost per request is (1 − hit rate) × cost of a call.

Why it matters

It makes the decision measurable.

Log a week of real requests, count how many repeat, and you know the hit rate before building anything.
If only two percent repeat, a response cache is not worth its risks.
If thirty percent do, it is the largest saving available.

In the real world: An analysis of a retailer's chat logs shows forty percent of questions are five questions asked in different words, which justifies a carefully restricted semantic cache.

Visual example

Computed

Monthly cost of 1,000,000 requests at different hit rates. Each hit is a call that is never made.

Per call: 3,000 tokens in, 300 out, medium tier (₹1.125 a call)

hit rate	average ₹ per request	monthly cost	monthly saving
0%	1.13	₹11.3 lakh	₹0
20%	0.9	₹9 lakh	₹2.3 lakh
40%	0.68	₹6.7 lakh	₹4.5 lakh
60%	0.45	₹4.5 lakh	₹6.7 lakh

Estimating the hit rate by intuition. It is usually either much lower than hoped (free-form chat) or much higher than expected (a few common questions dominating). Measure it on real logs.

Practise it: Design an LLM Inference Serving System

When not to cache

What it is

Some answers should never come from a response or semantic cache:

Personal: anything built from one user's data.
Live: order status, stock, prices, anything that changes by the minute.
Creative: where the user wants a different result each time.
Actions: a request that calls a tool must run the tool, not replay an old result.

Why it matters

A cache trades freshness and variety for speed and cost.

Where the answer must reflect the present moment or this particular person, that trade is wrong.
Prompt caching is the exception: it changes nothing about the answer, so it is safe everywhere.

In the real world: A delivery app caches 'what are your delivery hours' and never caches 'where is my order', whose answer is different every minute.

Caching in front of a tool-calling assistant. A cached 'your refund has been issued' replays the sentence without issuing the refund.

Practise it: Design an LLM Inference Serving System