LLM System Design · Module 16 of 17 · 8 concepts · about 12 min

Scaling and Capacity

How to work out the capacity a system needs, in tokens per minute, and the handful of ideas that govern speed and cost when the model runs on your own GPUs.

At work: The planning done before a launch or a big sale day, and the knowledge an interviewer probes when a design says 'we will self-host the model'.

Capacity planning in tokens per minute

What it is

Capacity for an LLM system is counted in tokens per minute (TPM), worked out in four steps:

average requests per second = monthly requests ÷ seconds in a month;
peak requests per second = average × a peak factor;
tokens per minute at peak = peak requests per second × 60 × tokens per request;
add headroom, usually half as much again.

Why it matters

Tokens per minute is the unit providers limit and the unit a GPU delivers, so it is the unit to plan in. The same request rate can mean very different token loads: a feature with long prompts needs many times the capacity of one with short prompts, at the same number of requests.

In the real world: Before a festival sale, a team works out its expected peak in tokens per minute, finds it is double its current allowance, and requests a quota increase three weeks ahead.

Visual example

Computed

Peak capacity needed for two of the running-example features: about 940,000 tokens a minute, before headroom.

tokens per minute = peak requests per second × 60 × tokens per request

feature	requests a month	average per second	peak factor	peak per second	tokens per request	tokens per minute
Support assistant	1,000,000	0.39	× 5	1.9	4,000	462,963
Order summaries	6,000,000	2.31	× 4	9.3	860	477,778
Total						940,741

With 1.5× headroom, ask for about 1,500,000 tokens a minute.

Planning for the average. Traffic is not flat: a grocery app's evening peak can be five times its daily average. A system sized for the average fails every evening.

Provider quotas and reserved capacity

What it is

A provider gives each customer a quota: a ceiling on requests and tokens per minute. There are two ways to buy capacity:

On demand: pay per token, share the provider's capacity with everyone, subject to your quota and to how busy they are.
Reserved (often called provisioned throughput): pay a fixed fee for capacity that is yours, with steadier speed.

Why it matters

On-demand capacity is cheapest at low and uneven volume, but its speed varies with the provider's load and your quota can be the thing that fails you at peak.

Reserved capacity costs the same whether used or not, like a server, and gives predictable latency.
Many systems reserve enough for their steady base load and let peaks spill over to on-demand.

In the real world: A company reserves capacity for its everyday chat volume and routes overflow at peak to on-demand, paying the fixed fee only for load it knows it will have.

Discovering the quota on launch day. Quota increases can take days or weeks to approve. Check the limits for every model in your fallback chain, not only the first choice.

Scaling your own services

What it is

Your own parts (the application, the gateway, the guardrails) scale the ordinary way:

they hold no state in memory, so any copy can serve any request;
conversation history, limits and caches live in shared stores;
more copies are added behind a load balancer as traffic grows.

One difference: a streaming request holds a connection open for many seconds, so each server handles fewer at once.

Why it matters

Because the model is a remote call and the conversation lives in a database, your services are easy to scale.

The limits that matter are elsewhere: the provider's tokens per minute, and the number of long-lived streaming connections each server can hold.
Capacity for the latter is counted in concurrent connections, not requests per second.

In the real world: A chat service that handled its load test easily falls over in production, because the test used short non-streamed requests and real users hold streams open for ten seconds each.

Sizing servers by requests per second, as for a normal API. A streaming chat request lasts five to twenty seconds. A hundred requests a second can mean a thousand or more connections open at once.

Prefill vs decode

What it is

When a model runs, each request has two phases with very different costs:

Prefill: the model reads the whole prompt in one pass. Fast per token, because all the tokens are processed together.
Decode: the model writes the answer one token at a time. Slow per token, because each new token needs its own pass.

Time to first token is mostly prefill; the rest of the wait is decode.

Why it matters

This is the reason output tokens are slower and dearer than input tokens.

To avoid re-reading everything for each new token, the model keeps the work it did on earlier tokens in memory: the KV cache.
That makes decoding fast, and it means every active request holds a block of GPU memory that grows with its length.
On a busy server, that memory, not raw compute, is often what runs out.

In the real world: A self-hosting team finds their server refusing new requests while its processor is half idle; the GPU's memory was full of the saved state of forty long-running conversations.

Visual example

Two phases

Reading is quick; writing is slow. Almost every serving decision follows from this.

The two phases of running one request

	prefill (reading the prompt)	decode (writing the answer)
Tokens handled	All input tokens, together	One output token at a time
Speed per token	Fast	Slow
Decides	Time to first token	Time to finish
Helped by	Shorter prompts, prompt caching	Shorter answers, a smaller model
Memory	Builds the KV cache	Reads and extends the KV cache

Assuming a GPU's limit is how fast it computes. With many long conversations in progress, it is usually memory for their KV caches that caps how many requests can run at once.

Practise it: Design an LLM Inference Serving System

Continuous batching

What it is

A GPU can process many requests at once nearly as fast as one. Batching runs requests together. Continuous batching does it without making anyone wait:

requests join the running batch as soon as they arrive;
each leaves as soon as its own answer is finished;
the batch is refilled at every step, so the GPU stays full.

Why it matters

A GPU serving one request at a time is mostly idle and extremely expensive per token.

Batching is what makes self-hosting affordable: total throughput rises many times over while each request slows only a little.
It is also why the same model is cheaper from a busy provider than from your own quiet server.
Serving software such as vLLM does this for you.

In the real world: A team's cost per token falls by a factor of ten when they move from a simple one-at-a-time server to one with continuous batching, on the same GPU.

Visual example

Illustrative

One GPU at three batch sizes. Each request slows slightly; the total served rises many times.

A model that writes 40 tokens a second for a single request (illustrative figures)

requests in the batch	tokens a second, each	tokens a second, total
1	40	40
8	35	281
32	25	790

The slowdown per request here is a simple model, not a measurement. The shape is what matters: batch for throughput, and watch the per-request speed.

Benchmarking with one request at a time. That measures the least efficient way to use the hardware and makes self-hosting look far more expensive than it is. Measure throughput at realistic concurrency.

Practise it: Design an LLM Inference Serving System

Quantization

What it is

A model is billions of numbers (its parameters), normally stored at 16 bits each. Quantization stores them with fewer bits:

8-bit halves the memory;
4-bit quarters it;
quality falls slightly, more so at very low precision.

Why it matters

Memory decides which GPU a model fits on, and so what it costs to run.

A model with 8 billion parameters needs about 16 GB at full size, plus room for the requests' saved state.
At 4-bit it needs about 4 GB, which fits a much cheaper card and leaves more memory for running requests at once.

In the real world: A team fits a model onto a single mid-range GPU by quantizing to 4-bit, measures a one-point drop on its test set, and accepts it for the saving.

Visual example

Computed

Memory for the weights of a model with 8 billion parameters, at three precisions.

memory = parameters × bytes per parameter

precision	bytes per parameter	memory for the weights (GB)
16-bit (the usual full size)	2	16
8-bit	1	8
4-bit	0.5	4

This is the weights alone. Running requests need more on top, for their saved state.

Quantizing and shipping without re-running the test set. The loss is usually small and occasionally not, and it varies by task. A model that is fine at 4-bit for chat may be noticeably worse at exact extraction.

Practise it: Design an LLM Inference Serving System

GPU autoscaling lag and load shedding

What it is

Adding GPU capacity is slow:

a new server has to be found and started;
many gigabytes of model weights have to be loaded into GPU memory;
in all, several minutes, where a web server takes seconds.

This delay is autoscaling lag. A traffic spike arrives faster than new capacity can.

Why it matters

For the minutes it takes capacity to arrive, demand exceeds supply.

Without a plan, every request slows and many time out together.
So a self-hosted system keeps spare capacity warm for its expected peaks, and for the unexpected ones it sheds load: it refuses or downgrades low-priority requests quickly, so that the rest are still served on time.

In the real world: A self-hosted assistant keeps two idle GPUs warm every evening before the dinner rush, because scaling up from cold took six minutes and the rush builds in two.

Relying on autoscaling alone, as you would for a web tier. By the time a new GPU is serving, the spike may be over, and everything that arrived in between has been slow or lost.

Practise it: Design an LLM Inference Serving System

Sharing capacity fairly between tenants

What it is

When many teams or customers share one pool of model capacity, each needs protecting from the others:

a limit per tenant, in tokens per minute;
priorities: live customer traffic ahead of batch work;
fair queuing: when capacity is short, each tenant gets its share in turn, however much it is asking for.

Why it matters

A shared pool is efficient, because tenants peak at different times.

It is also where one tenant's runaway job can take everyone down.
Per-tenant limits and priorities turn a shared outage into a problem only for the tenant causing it.
This is the same idea as the gateway's rate limits, applied to capacity that is fixed.

In the real world: An internal platform gives each team a guaranteed share and lets them borrow idle capacity; a team's overnight experiment can no longer slow the customer chat.

Capacity: 1,000,000 tokens a minute, three tenants

  Tenant        asks for     fair share    gets
  Support        300,000      333,333      300,000
  Summaries      250,000      333,333      250,000
  Batch job    2,000,000      333,333      450,000   <- the rest, no more

The batch job cannot crowd out the other two.

First come, first served. Whichever tenant sends the most requests then gets the most capacity, which rewards exactly the behaviour you want to contain.