The LLM System Design Interview

Answer a design question in a fixed order:

• 1. Clarify the requirements: users, volume, latency, quality, cost, safety.
• 2. Estimate the scale: requests, tokens per minute, cost.
• 3. Sketch the request path: the eight stages from application to logging.
• 4. Go deep on the two or three decisions that matter most for this system.
• 5. Reliability: what happens when the model is slow, down or rate-limited.
• 6. Safety and evaluation: guardrails, test sets, release gates.
• 7. Operations: cost tracking, monitoring, how changes are rolled out.

Interviewers grade structure as much as content.

• A candidate who starts by naming a model has skipped the questions that decide which model is right.
• Following the order shows you can run a project, and it guarantees you reach reliability, safety and evaluation, the three areas most often left out and the ones senior interviewers care about most.

The questions to ask before drawing anything:

• Who uses it, and how much? Users, requests per day, the peak.
• Is someone waiting? A live chat, or work that can run later?
• How fast? A target for the first word and for the whole answer.
• How good must it be? What does a wrong answer cost?
• What may it do? Only answer, or also take actions?
• What data is involved? Personal data, rules about where it may go.
• What is the budget? Per request or per month.

Each answer changes the design.

• A live chat means streaming and a tight latency budget.
• Actions mean tool calling with permission checks. Personal data may rule out some providers.
• A high cost of error means verification and a route to a human.
• Asking shows you know which facts drive which decisions.

Five quick estimates, in order:

• requests per day = users × messages per user;
• average and peak requests per second;
• tokens per minute at peak = peak requests per second × 60 × tokens per request;
• streams open at once = peak requests per second × seconds per answer;
• cost per day = requests × cost per request.

Estimates turn opinions into decisions.

• Tokens per minute says whether one provider's quota is enough. Open streams sizes your own servers.
• Cost per day says whether the largest model is affordable, and usually shows it is not, which leads naturally to routing and caching.
• Being roughly right, quickly, with the working shown, is what is being tested.

Most choices in an LLM system trade two things, and interviewers want both sides named:

• Model size: quality against cost and latency.
• Context size: more information against cost, speed and focus.
• Streaming: perceived speed against the ability to check the whole answer first.
• Caching: speed and cost against freshness.
• Guardrails: safety against latency and blocked legitimate requests.
• Retries: reliability against added delay and load.
• Hosted or self-hosted: simplicity against control and cost at scale.
• Autonomy: usefulness against the risk of wrong actions.

A choice stated without its cost sounds like a habit.

• A choice stated with its cost, and with the measurement that would settle it, sounds like engineering.
• The pattern for any trade-off answer is: the two sides, what you would pick by default, and what you would measure to check.

The most common question. The points an interviewer wants to hear, in order:

• Streaming over server-sent events, with cancellation when the user leaves.
• Conversations stored per message with a parent, so edits and regenerations form a tree.
• Context management: a token budget, recent turns in full, older ones summarised.
• Prompt caching for the unchanging start of the prompt.
• Model tiering: a small model for titles and easy turns, a larger one for hard turns.
• Guardrails on input and on the streamed output.
• Fallback to another provider on timeout, behind a gateway.
• Limits per user, in tokens, and a cost budget.

Each point answers a follow-up the interviewer is likely to ask: how does the user see words appear, what happens to a long chat, what happens when the provider is down, how is one abusive user stopped. Stating them in this order shows the request path, then the state, then the failure handling, which is the natural shape of the system.

The platform version of the question. The points to cover:

• One request format for callers, with an adapter per provider.
• Model aliases, mapped to pinned versions, with weighted rollout.
• Virtual keys per team, holding limits and budgets; provider keys stay inside.
• Rate limits in tokens per minute, with a token bucket in a shared store.
• Reliability: timeouts, retries with backoff and jitter, circuit breakers, fallback chains.
• Streaming passed through and normalised across providers.
• Metering: tokens and cost per team and feature, from the provider's own counts.
• Logging off the request path, with rules for sensitive content.
• The gateway itself: stateless copies, a few milliseconds of overhead, fails open where safe.

This question tests whether you see the gateway as a reliability and cost product, not a simple proxy. The interviewer will press on its failure modes: what happens if the limit store is down, if a provider streams a malformed chunk, if a budget runs out mid-conversation. Have an answer for each.

The questions that open most interviews. They test whether you can explain the basics plainly and reason about money.

A crisp answer here earns the right to the harder questions. Each answer below is the two or three sentences an interviewer is listening for; the detail is in the earlier modules.

The middle of the interview, where the interviewer checks that you have run one of these under real load.

These are the questions with the most room to show depth, so tie each answer to a failure it prevents.

The questions that distinguish someone who has shipped an LLM feature from someone who has built a demo.

Senior interviews spend most of their time here. The answers come back to two habits: enforce what matters in code, and measure before and after every change.

Statements that signal a candidate has only built a demo:

• Naming a model first, before asking what the system is for.
• The largest model for everything, with no mention of cost.
• No streaming for a live chat.
• 'The prompt will tell it not to' as a safety measure.
• No plan for provider failure.
• No evaluation, or 'we will try some examples'.
• No numbers: no token, capacity or cost estimate.
• Every technique at once, with no reason for any of them.

Each has a positive counterpart, and together they describe a strong answer: requirements first, the smallest model that works, streaming with a latency budget, enforcement in code, a fallback chain, a test set with a release gate, estimates in tokens and rupees, and complexity added only where a stated need justifies it.