LLM System Design · Module 12 of 17 · 7 concepts · about 12 min

Reliability

Model providers slow down, refuse requests and go offline. This module covers how calls fail, and the techniques that keep a product working anyway: retries, fallbacks, circuit breakers, queues and graceful degradation.

At work: What an on-call engineer relies on at dinner time when the provider's status page turns orange and the support chat still has to answer.

How model calls fail

What it is

A model call can fail in more ways than an ordinary service call:

Rate limited (HTTP 429): you are over your per-minute allowance.
Overloaded or erroring (HTTP 5xx): the provider is having trouble.
Timed out: no answer, or no first token, in the time allowed.
Refused: the provider's content filter blocked the request or the answer.
Malformed: an answer arrived but is cut off or not in the format asked for.
Wrong: a fluent, well-formed, incorrect answer, with no error at all.

Why it matters

Each kind needs a different response.

Retrying a rate limit immediately makes it worse; retrying a refusal never works; a malformed answer may be fixed by asking again; a wrong answer is not detected by any of this and needs the checks in the Guardrails and Evaluation modules.
A design that treats 'the call failed' as one case handles most of them badly.

In the real world: An incident review finds that a storm of blind retries during a brief provider slowdown tripled the traffic and turned a two-minute blip into a twenty-minute outage.

Visual example

Failures

The first question is always: which kind is it?

Each failure and the right response

failure	retry?	what to do
Rate limited (429)	Yes, after waiting	Back off; use another route or model
Overloaded / server error (5xx)	Yes, a few times	Back off, then fall back
Timed out	Once, or fall back	Switch model; do not wait again
Refused by a content filter	No	Tell the user; do not retry
Malformed output	Yes, with the error	Ask again; then hand to a person
Invalid request (4xx)	No	A bug on your side; fix it

Catching every exception and retrying. That hammers a rate-limited provider, repeats a request that will always be refused, and can double an action that actually succeeded. Classify the failure first.

Retries with backoff and jitter

What it is

A retry tries a failed call again. Done properly:

Exponential backoff: wait longer before each attempt, typically doubling.
Jitter: add a random amount to each wait, so many clients do not retry at the same instant.
A cap on the number of attempts and on the longest wait.
If the provider says how long to wait (a Retry-After header), use that.

Why it matters

Many failures are brief, and a second attempt a moment later succeeds.

But if every client retries at once, on the same schedule, the retries arrive as a synchronised wave and knock over a provider that was just recovering.
Backoff spreads the load over time; jitter spreads it between clients.

In the real world: After adding jitter, a team's recovery from provider blips changes from a second wave of failures to a smooth return, with no other change.

Visual example

Computed

The longest wait before each retry: 500 ms, doubling, capped at 8,000 ms. With jitter, the real wait is a random time up to this.

wait = min(cap, base × 2^attempt)

retry	longest wait (ms)	total waited so far (ms)
1	500	500
2	1000	1500
3	2000	3500
4	4000	7500

Four retries can add up to 7.5 seconds of waiting. For a live chat that is usually too long, which is why falling back to another model comes before the last retries.

import random, time

def call_with_retries(request, attempts: int = 4, base: float = 0.5, cap: float = 8.0):
    for attempt in range(attempts + 1):
        try:
            return provider.chat(request)
        except RetryableError as error:
            if attempt == attempts:
                raise
            wait = min(cap, base * 2 ** attempt)
            wait = error.retry_after or random.uniform(0, wait)   # full jitter
            time.sleep(wait)

Retrying at every layer. If the application retries three times, the gateway three times and the provider's library three times, one failure becomes twenty-seven calls. Retry in one place, usually the gateway, and nowhere else.

Provider fallback

What it is

Provider fallback means that when the first-choice model fails, the request is sent to another. A fallback chain lists the alternatives in order:

the same model at another region or account;
an equivalent model from a second provider;
a smaller, cheaper model that is nearly always available.

The gateway walks down the chain until one answers.

Why it matters

A retry helps with a brief blip.

It does nothing when a provider is down for an hour.
A fallback to an independent provider keeps the product working through an outage that would otherwise be total.
The second model may give somewhat different answers, which is why the prompts must have been tested against it in advance.

In the real world: A retailer keeps two percent of its chat traffic permanently on its backup provider, so that on the day the primary fails, the backup is known to work and its quota is already warm.

aliases:
  support-chat:
    chain:
      - provider-a/medium        # first choice
      - provider-a-eu/medium     # same model, another region
      - provider-b/medium        # a different provider
      - provider-b/small         # degraded, but answers
    first_token_timeout_ms: 5000

A fallback that has never been used. If the second provider's model has not been tested with your prompts, its rate limits have not been raised, and its key has expired, the fallback fails at exactly the moment it is needed. Send a small share of real traffic down it all the time.

Practise it: Design a ChatGPT-like Application

Circuit breaker

What it is

A circuit breaker stops sending requests to a route that is clearly failing. It has three states:

Closed: normal; requests flow, and failures are counted.
Open: too many recent failures; requests skip this route and go straight to the fallback.
Half-open: after a pause, a few test requests are let through to see whether it has recovered.

Why it matters

Without it, every request during an outage first waits for the failing provider to time out, and only then falls back.

Users wait five extra seconds each, and the struggling provider is kept under load.
With a breaker, the failing route is skipped at once and is given room to recover.

In the real world: When one region of a provider degrades, its breaker opens in seconds and traffic moves to another region; most users notice nothing.

Visual example

Circuit breaker

It moves between them on its own, from the failures it sees.

The three states of a circuit breaker

ClosedRequests flow; failures counted
OpenToo many failures: use the fallback
Half-openAfter a pause, a few test requests
Closed againTests pass; if not, back to open

class CircuitBreaker:
    def __init__(self, max_failures: int = 5, cooldown: float = 30.0):
        self.max_failures, self.cooldown = max_failures, cooldown
        self.failures, self.opened_at = 0, None

    def allow(self) -> bool:
        if self.opened_at is None:
            return True                                   # closed
        return time.monotonic() - self.opened_at > self.cooldown   # half-open

    def record(self, ok: bool) -> None:
        if ok:
            self.failures, self.opened_at = 0, None
            return
        self.failures += 1
        if self.failures >= self.max_failures:
            self.opened_at = time.monotonic()             # open

One breaker for a whole provider. A provider can be failing for one model or one region and healthy for the rest. Keep a breaker per route (provider, model and region), so that one bad route does not take the healthy ones out of use.

Practise it: Design an LLM Gateway

Graceful degradation

What it is

Graceful degradation means offering something less good in place of an error. In order of preference:

a smaller model, with somewhat weaker answers;
a cached answer, marked as possibly out of date;
the feature without AI: a search results page, a form, a set of help links;
a clear message and a way to reach a person.

Why it matters

For most features, a weaker answer or a plain non-AI fallback is far better than a failure.

A support chat that says 'I'm having trouble, here are the three most relevant help articles and a button to reach an agent' has kept the customer.
One that shows a spinner for thirty seconds and then an error has lost them.

In the real world: During a provider outage, an airline's assistant switches to showing the matching help articles with a note that the assistant is temporarily limited, and call-centre volume barely rises.

Degrading silently in ways the user should know about. Serving a stale cached answer about an order's status as if it were current is worse than an honest 'I can't check that right now'. Degrade openly where correctness matters.

Practise it: Design an LLM Gateway

Queues and load shedding

What it is

Two ways to cope when there is more work than capacity:

A queue holds work that can wait and feeds it to the model at a steady rate. It suits anything nobody is watching: batch jobs, summaries, emails.
Load shedding deliberately refuses or downgrades some requests so the rest are served well. Low-priority traffic is shed first.

Why it matters

Capacity is a fixed number of tokens per minute.

When demand exceeds it, something must give.
Without a plan, every request slows down and many time out together.
A queue moves non-urgent work out of the peak; shedding protects the requests that matter, such as a paying customer's live chat, at the expense of those that do not, such as a background re-indexing job.

In the real world: At the evening peak a platform pauses its batch description jobs automatically and resumes them after midnight, keeping the whole rate limit for customers.

Visual example

Overload

Decide the priority order in advance, not during the incident.

What to do with each kind of work when capacity runs out

kind of work	example	under overload
Live, customer-facing	Support chat	Serve first; degrade to a smaller model
Live, internal	An employee tool	Slow down; queue briefly
Can wait minutes	Order summaries	Queue
Can wait hours	Product descriptions	Pause; run off-peak

Queueing live chat. A queue in front of a person who is waiting is just a longer wait, and by the time the request is served the user has gone. Queues are for work with no one watching; for live traffic, shed or degrade.

Practise it: Design an LLM Inference Serving System

Availability with more than one provider

What it is

Availability is the share of time a service works. If a single provider is up a certain share of the time, then two independent providers, with fallback between them, are both down only when each is down at once:

availability of the pair = 1 − (1 − a)², for two providers each available a share a of the time.

Why it matters

The effect is large.

One provider at 99% is down about 7.2 hours a month.
Two such providers, failing independently, are both down about 4 minutes a month.
This is the arithmetic behind the advice to have a second provider, and it is worth being able to do in an interview.

In the real world: A payments company's reliability review rejects a 'fallback' that was a second account with the same provider, since both would have failed in the same outage.

Visual example

Computed

One provider against two independent ones, each up 99% of the time. The pair is down a small fraction as often.

Availability over a 720-hour month

setup	availability	down per month
One provider	99%	7.2 hours
Two independent providers, with fallback	99.99%	4 minutes

The arithmetic

one down: 1 − 0.99 = 0.01

both down: 0.01 × 0.01 = 0.0001

pair up: 1 − 0.0001 = 99.99%

Assuming the two are independent when they are not. Two models served from the same cloud region, or two accounts with the same provider, fail together. The calculation only holds for genuinely separate providers.