LLM System Design · Module 9 of 17 · 7 concepts · about 11 min

Latency and Streaming

Where the waiting time goes in an LLM request, how streaming makes a slow answer feel fast, and the handful of levers that actually shorten it.

At work: What decides whether a chat feature feels alive or broken. Users forgive a long answer that starts at once far more readily than a short one that starts late.

The latency budget

What it is

A latency budget splits the time a user is willing to wait across the stages of a request. Two targets are set:

time to first word: how long until the answer starts to appear;
time to complete: how long until it finishes.

Every stage before the model call, and the model's own start-up time, spends from the first.

Why it matters

Writing the budget down shows where the time really goes and stops effort being wasted on the wrong stage.

In most systems the model's writing time dwarfs everything else, and the stages before it matter mainly because they delay the first word.
Targets are stated as a percentile such as p95 (95 percent of requests are faster than this), because the average hides the slow requests users remember.

In the real world: A team sets a target of 'first word within one second for 95 percent of chats', measures each stage, and finds a sequential database lookup was costing a third of that budget.

Visual example

Computed

One support request on the medium tier. The first word arrives at 815 ms; the rest of the answer takes far longer to write.

Where the time goes (illustrative stage times; 300-token answer)

stage	time (ms)	note
Input guardrails	80	A fast classifier, not a large model
Build context (three lookups, in parallel)	120	Would be 210 ms one after another
Build the prompt, pass the gateway	15	Should be near nothing
Model: time to first token	600	Shorter prompt, smaller model, prompt caching
Time to first word	815	What the user feels
Writing 300 tokens at 80 a second	3750	Shorter answers, a faster model
Time to complete	4565

Reporting the average. An average of two seconds can hide one request in twenty taking fifteen. Users judge a product by its bad moments, so design and measure for p95 or p99.

TTFT and how speed is perceived

What it is

TTFT is time to first token: the delay before the first piece of the answer appears. What users perceive depends on three moments:

something happens (a typing indicator): should be immediate;
the first words appear: TTFT plus everything before the model call;
the answer finishes: TTFT plus the writing time.

Why it matters

People read at a few words per second, and a model writes faster than that.

So once the first words are on screen, the user is occupied, and the rest of the wait is hidden behind their own reading.
This is why a four-second answer that starts in half a second feels faster than a two-second answer that appears all at once.

In the real world: A product team A/B tests two versions with the same total time; the one that shows its first words a second sooner is rated noticeably faster by users.

Optimising total time for a chat interface. For anything a person reads as it arrives, the first word is the number to defend. Total time matters most when a program, not a person, is waiting for the whole result.

Practise it: Design an LLM Inference Serving System

Server-sent events streaming

What it is

Streaming sends the answer to the user piece by piece as the model writes it. The usual mechanism is server-sent events (SSE):

the browser opens one ordinary HTTP request and keeps it open;
the server writes small text events down it, one per chunk of the answer;
a final event says the answer is complete, and carries the token counts.

Why it matters

SSE is simple, works over plain HTTP, passes through most proxies and reconnects on its own.

It is one-way, server to client, which is all a streamed answer needs.
A WebSocket, which is two-way, is only worth its extra complexity when the client also sends a stream, as in live voice.

In the real world: A team's streaming works on a developer's laptop and arrives all at once in production; the cause is the load balancer buffering responses, fixed with one configuration line.

from fastapi.responses import StreamingResponse

def events(messages):
    for chunk in gateway.stream(model="support-chat", messages=messages):
        yield f"data: {json.dumps({'text': chunk.text})}\n\n"
    yield "event: done\ndata: {}\n\n"

@app.post("/chat")
def chat(request: ChatRequest):
    return StreamingResponse(events(request.messages), media_type="text/event-stream")

A proxy or load balancer that buffers the response. Many are set up to collect a whole response before forwarding it, which silently turns your stream back into one long wait. Buffering has to be switched off for streaming routes at every hop.

Practise it: Design a ChatGPT-like Application

Streaming through the whole stack

What it is

A stream passes through several of your own services, and each must handle it properly:

the gateway forwards chunks as they arrive and normalises each provider's chunk format;
cancellation flows backwards: if the user closes the tab, the request to the provider is cancelled;
errors mid-stream are reported as an event, since the HTTP status was already sent as success;
token usage is captured from the final chunk for metering.

Why it matters

Cancellation is the one that costs money.

A user who leaves after two words still triggers a full answer unless the disconnect is passed along, and you pay for every token of an answer nobody reads.
Mid-stream errors are the one that confuses users: half an answer and silence looks like a finished, wrong answer unless the client is told otherwise.

In the real world: A provider's bill falls by several percent after a company starts cancelling upstream requests when users navigate away mid-answer.

Saving the message to the database only when the stream completes normally. If the user disconnects or an error occurs halfway, the partial answer the user saw is not recorded, and the next turn's history does not match what was on their screen.

Practise it: Design a ChatGPT-like Application

Doing work in parallel

What it is

Steps that do not depend on each other should run at the same time:

loading history, retrieving documents and looking up the order can start together;
an input safety check can run alongside context building;
independent sub-questions can be sent to the model at once.

The wait is then the slowest of the group, not their sum.

Why it matters

In the budget above, the three context lookups take 30, 120, 60 ms.

One after another that is 210 ms; together it is 120 ms.
The same idea applies to model calls: three independent checks made in parallel cost the same money as three made in sequence and take a third of the time.

In the real world: A support assistant's time to first word drops by two hundred milliseconds when its three lookups, written one after another for no particular reason, are started together.

import asyncio

async def build_context(user, question):
    history, passages, order = await asyncio.gather(
        load_history(user),            #  30 ms
        retrieve(question),            # 120 ms
        lookup_order(user, question),  #  60 ms
    )                                  # total: 120 ms, not 210
    return history, passages, order

Chaining model calls that could have been one. Rewriting the question, then classifying it, then answering, as three calls in a row, triples the wait. Ask whether one call with a good prompt, or two calls in parallel, would do.

The levers that reduce latency

What it is

In rough order of effect:

Stream the answer, so the wait is hidden behind reading.
Ask for less: a shorter answer is the largest single saving in total time.
Use a smaller model where it is good enough.
Cache repeated requests and the unchanging start of the prompt.
Parallelise the steps before the call.
Remove steps: every extra model call before the answer adds its full duration.

Why it matters

Output length and model size dominate.

On the example tiers, a 300-token answer takes 4.4 seconds on the medium model and 2.3 on the small one; a 100-token answer on the medium model takes 1.9.
Shaving 20 ms off a database query is invisible next to either.

In the real world: An order-summary feature goes from three seconds to under one by moving to a small model and capping the summary at two sentences; no infrastructure changed.

Visual example

Levers

Start at the top. The first three are usually enough.

What each lever improves

lever	first word	total time	cost
Stream the answer	Feels immediate	Unchanged	Unchanged
Ask for a shorter answer	Unchanged	Much lower	Lower
Smaller model	Lower	Much lower	Much lower
Cache	Near zero on a hit	Near zero on a hit	Lower
Parallel steps before the call	Lower	Slightly lower	Unchanged
Fewer model calls in a row	Lower	Lower	Lower

Tuning infrastructure first. Teams spend a week on connection pooling and network hops when the answer was twice as long as it needed to be. Measure, then start with the largest number.

Timeouts and deadlines

What it is

Every call has a timeout: a limit after which you stop waiting. For a streamed model call there are two:

a first-token timeout: nothing has arrived after a few seconds, so give up and try elsewhere;
an overall timeout: the whole answer is taking too long.

A deadline is the total time the user's request is allowed, shared by every step inside it.

Why it matters

A model call that hangs holds a connection, a worker and the user.

The first-token timeout is the useful one: a healthy call starts within a second or two, so silence for five is a strong sign of trouble, and switching to another model then is far better than waiting thirty.
A shared deadline stops retries deep in the system from running long after the user has given up.

In the real world: During a provider slowdown, a five-second first-token timeout moves traffic to a backup model automatically; users see slightly slower answers and never an error.

Request deadline: 20 seconds

  first-token timeout   5 s   -> no token by then: fall back to another model
  overall timeout      20 s   -> cut off, return what has been written

A retry started at second 18 gets 2 seconds, not a fresh 20.

Copying a web service's timeout of a second or two. Model calls legitimately take many seconds. Too short a limit cuts off good answers; none at all lets a stuck call hang for minutes. Set it from your measured p99.