LLM System Design · Module 1 of 17 · 7 concepts · about 11 min

What LLM System Design Is

What is different about a system built around a language model, the parts every such system has, and the questions an interviewer expects you to ask before drawing anything.

About LLM System Design: A language model on its own is a single call: text goes in, text comes out. Turning that into a product people rely on is a different job, because the call is slow, costs money for every word, sometimes fails and is sometimes wrong. LLM system design is how you build around it. You will start with what makes these systems different and how one model call works, then build each part in turn: choosing a model, the gateway that every request passes through, prompts, memory, tools, speed, caching, cost, reliability and safety. After that you will learn how to measure quality, how to run the system once it is live, and how to answer the questions interviewers ask about all of it.

At work: The work of an AI platform or product engineering team: turning a model that answers in a demo into a product that serves many users quickly, cheaply and safely.

What an LLM system is

What it is

A large language model (LLM) takes text in and writes text out. An LLM system is everything you build around that one call so that it works as a product.

Before the call: checking the request, finding the right information, and building the prompt.
The call itself: choosing a model, sending the request, and handling slow or failed responses.
After the call: checking the answer, formatting it, logging it, and paying for it.

Why it matters

The model is a small part of the work. A demo is one prompt and one answer. A product has to handle:

thousands of users at once;
a provider that is sometimes slow or down;
a bill that grows with every word;
users who try to misuse it;
answers that are sometimes wrong.

LLM system design is the discipline of handling those five things on purpose.

In the real world: A food-delivery company's support chatbot works in the demo, then fails in its first week: answers take 12 seconds at dinner time and the monthly bill is five times the estimate.

Thinking the interview is about the model. Interviewers rarely ask how a transformer works in a system design round. They ask what happens when the model is slow, wrong, expensive or attacked.

Practise it: Design an LLM Inference Serving System

How LLM systems differ from ordinary software

What it is

Five properties of a model call are unlike a normal function or database call:

It is slow: seconds, not milliseconds, and the time grows with the length of the answer.
It is priced per word: every token in and out costs money, on every request.
It is not repeatable: the same question can get a different answer each time.
It can be confidently wrong: there is no error code for a made-up fact.
It follows text: instructions hidden in its input can change what it does.

Why it matters

Each property forces a design decision that ordinary systems do not need:

Slow leads to streaming, caching and time limits.
Priced per word leads to budgets, smaller models and shorter prompts.
Not repeatable leads to evaluation with test sets, not single checks.
Confidently wrong leads to grounding, verification and human handover.
Follows text leads to guardrails and strict limits on what the model may do.

In the real world: An engineer used to databases sets a 500 millisecond timeout on the model call, and every long answer is cut off, because the call routinely takes four seconds.

Visual example

Comparison

The same request-and-response shape, with very different behaviour.

A database call versus a model call

property	database call	model call
Typical time	Milliseconds	Seconds
Cost per call	Close to nothing	Real money, by the token
Same input, same output?	Yes	Not guaranteed
How it fails	An error you can catch	A fluent wrong answer
Can its input change its behaviour?	No	Yes (prompt injection)
Capacity limit	Queries per second	Tokens per minute

Designing it like a normal web service and adding 'call the LLM' as one box. The interviewer's follow-ups will all be about the five properties above, and a design that ignored them has no answers.

The parts of an LLM application

What it is

Most production systems have the same parts, in the same order. A request passes through:

the application (the chat screen or feature the user sees);
input guardrails (is this request allowed and safe?);
context building (history, retrieved documents, user details);
the prompt template (instructions plus that context);
the gateway (which model, which provider, with retries and limits);
the model itself;
output guardrails (is this answer safe and well-formed?);
logging and evaluation (what happened, what it cost, was it good?).

Why it matters

This list is the skeleton of every design answer.

Each later module fills in one part.
In an interview, drawing these eight boxes first gives you a map to hang every follow-up question on.

In the real world: A bank's architecture review asks one question of every new AI feature: which of these eight parts are you using from the shared platform, and which are you building yourself?

Visual example

Reference design

Eight stages. Every later module in this topic is about one of them.

One request through an LLM application

Application“Where is my refund?”
Input guardrailsAllowed, safe, in scope?
ContextHistory, the order, the policy
Prompt templateInstructions + context + question
GatewayPick a model; limits; fall back
ModelAnswer, streamed word by word
Output guardrailsChecked before it is all shown
Log and evaluateTokens, time, cost, quality

Letting each feature call the model provider directly. Then every team re-solves retries, limits, logging and cost tracking on its own, and none of them do it well. The shared parts belong in one place.

The running example: Kirana Kart's AI platform

What it is

Kirana Kart, a grocery delivery app, now has three features that call a language model:

a support assistant that chats with customers about orders and refunds;
order summaries written for delivery partners;
product descriptions generated overnight for new items in the catalogue.

One platform team has to serve all three.

Why it matters

The three features want different things, which is what makes the design interesting:

The support assistant is live: it needs speed, safety and high quality.
Order summaries are high-volume and simple: they need to be cheap.
Product descriptions are offline: nobody is waiting, so they can be slow and batched.

One model and one set of settings would be wrong for at least two of them.

In the real world: An online marketplace runs its buyer chat on a fast, strong model, and its nightly listing clean-up on a small one through a discounted batch service, at a fraction of the cost per item.

Visual example

Workloads

The same platform serves all three, with different choices for each.

Three features, three sets of needs

	support assistant	order summaries	product descriptions
Is a person waiting?	Yes, in a chat	Briefly	No
Volume	Medium	Very high	Batch, overnight
What matters most	Quality and safety	Cost	Cost and throughput
Model	Strong	Small	Small, batched
Streaming	Yes	No	No

Designing for one workload and assuming the rest are the same. The first clarifying question in any design is: who is waiting for this answer, and for how long?

Requirements that drive the design

What it is

Before designing, pin down six things:

Users and volume: how many requests per day, and at the busiest minute?
Latency: how fast must the first word and the full answer arrive?
Quality: what does a wrong answer cost?
Cost: what is the budget per request or per month?
Safety and privacy: what data is involved, and what must never happen?
Reliability: what should the user see when the model is unavailable?

Why it matters

Each answer removes options.

A two-second limit rules out long chains of model calls.
A tight budget rules out the largest model for every request.
Personal data may rule out sending text to an outside provider at all.
Asking shows that you know which facts decide which choices.

In the real world: Told that 'agents must be able to answer in the customer's own language', a strong candidate immediately adds language to the model choice and to the test set.

Visual example

Requirements

Ask the question, then state the consequence.

Each requirement and what it decides

ask about	it decides
Requests per day and at peak	Capacity, rate limits, cost
Time to first word and to full answer	Streaming, model size, how many steps you can afford
Cost of a wrong answer	Verification, human handover, evaluation depth
Budget	Model tier, caching, prompt length
Kind of data	Which providers are allowed, what may be logged
Behaviour when the model is down	Fallback models, queues, a graceful message

Asking the questions and then ignoring the answers. After each answer, say what it changes: 'the first word must appear in one second, so I will stream and keep every step before the model call short.'

The four forces: quality, latency, cost, safety

What it is

Almost every decision in an LLM system trades four things against each other:

Quality: how good and how correct the answers are.
Latency: how long the user waits.
Cost: how much each request costs.
Safety: how well the system resists misuse and avoids harm.

Why it matters

Improving one usually costs another:

A larger model raises quality, latency and cost together.
A safety check before and after the call adds latency and cost.
A cache cuts latency and cost, and risks a stale answer.
A shorter prompt cuts cost and may cut quality.

A good design says which force matters most for this feature and what it gives up for it.

In the real world: A product manager wants the assistant 'smarter, faster and cheaper'; the engineer shows the three are linked and asks which one matters most for this quarter.

Visual example

Trade-offs

↑ goes up, ↓ goes down. Nothing improves all four.

What each common change does to the four forces

change	quality	latency	cost	safety
Use a larger model	↑	↑ (worse)	↑ (worse)	=
Add output checks	=	↑ (worse)	↑ (worse)	↑
Cache answers	risk of stale	↓	↓	=
Shorten the prompt	may ↓	↓	↓	=
Route easy requests to a small model	small risk	↓	↓	=

Claiming a design is the best on all four. That tells the interviewer you have not noticed the trade. Name the one you are favouring and the price you are paying.

Practise it: Design a ChatGPT-like Application, Design an LLM Gateway, Design an LLM Inference Serving System

A model call in code

What it is

Underneath every feature is the same small call: a list of messages goes in, text and a token count come out. The example uses a placeholder client standing for whichever provider you call; the shape is the same across all of them.

Why it matters

Three details in this small example come back in every later module:

The system message holds your instructions, separate from what the user typed.
The reply carries usage: input and output token counts, which are what you are billed for.
The call can fail or take a long time, so it is wrapped in a timeout.

In the real world: A team's first production incident is a single request that asked for 'a complete history of everything' and ran for four minutes; an output cap was added that afternoon.

reply = client.chat(
    model="medium",
    messages=[
        {"role": "system", "content": "You are Kirana Kart support. Be brief."},
        {"role": "user", "content": "Where is my refund for order 88213?"},
    ],
    max_output_tokens=300,     # a cap on length, cost and time
    timeout=20,                # seconds; never wait for ever
)

reply.text                      # the answer
reply.usage.input_tokens        # what you sent, in tokens
reply.usage.output_tokens       # what it wrote, in tokens

Leaving out the output cap and the timeout. Without them, one unusual request can produce a very long, very slow, very expensive answer, and nothing stops it.