LLM System Design · Module 4 of 17 · 8 concepts · about 13 min

The LLM Gateway

One service that sits between every feature and every model provider. It is where access, limits, cost tracking and logging are handled once, for everyone.

At work: The first thing a platform team builds once a company has more than one AI feature: a single front door to the models, so that each product team does not rebuild the same plumbing.

Why a gateway exists

What it is

An LLM gateway is a service of your own that every feature calls, and which in turn calls the model providers. It handles, in one place:

which model and provider a request goes to;
who is allowed to call, and how much;
retries, time limits and fallbacks when a provider fails;
counting tokens and cost per team and feature;
logging every request for debugging and evaluation.

Why it matters

Without it, each team calls providers directly and solves those five problems separately, each a little differently and none completely.

Provider keys end up scattered through many codebases, nobody can say what the company spends on AI, and switching model means changing every application.
A gateway turns five problems solved badly ten times into five problems solved well once.

In the real world: A company with nine teams calling models discovers it holds forty provider keys and cannot attribute a rising bill to any team; a gateway is the first item on the next quarter's plan.

Visual example

Gateway

Every feature's request takes the same path. None of this is rebuilt per team.

What the gateway does to each request

AuthenticateWhich team and feature?
Check limitsRate limit and budget
Resolve the modelName → real model and provider
Call the providerTime limit; retry or fall back
MeterTokens and cost
LogRequest, response, time, cost

Building a gateway before there is a second feature. For one small application, calling the provider directly is fine. The gateway earns its keep when several teams, several models or real cost and reliability pressure appear.

Provider abstraction layer

What it is

Each provider has its own request format, its own way of streaming and its own error codes. The provider abstraction layer is the part of the gateway that hides those differences:

applications send one standard request shape;
an adapter per provider translates it into that provider's format;
responses, streamed pieces, token counts and errors are translated back into one standard shape.

Why it matters

It is what makes providers swappable.

Application code is written once against your format.
Adding a provider means writing one adapter, not changing every application.
It also normalises the awkward parts that differ most: how streamed chunks arrive, how tool calls are expressed and how 'rate limited' or 'overloaded' is reported.

In the real world: A team adds a newly released model from a second provider in two days by writing one adapter; no product team changes a line of code.

class Provider:                     # one of these per provider
    def chat(self, request: ChatRequest) -> ChatResponse: ...
    def stream(self, request: ChatRequest) -> Iterator[Chunk]: ...

PROVIDERS = {
    "provider-a": ProviderA(),      # translates to and from A's format
    "provider-b": ProviderB(),
    "self-hosted": OwnCluster(),
}

def call(model: Model, request: ChatRequest) -> ChatResponse:
    return PROVIDERS[model.provider].chat(request)

Reducing every provider to the features they all share. If one provider has a valuable feature such as prompt caching, the abstraction should let a caller use it where available, not hide it from everyone.

Practise it: Design an LLM Gateway

Model aliasing

What it is

With model aliasing, applications never name a real model. They ask for an alias that describes a purpose, and the gateway maps it to a real model:

support-chat might map to a medium model at provider A;
cheap-fast to a small model;
best-reasoning to the largest available.

The mapping lives in the gateway's configuration and can be changed without touching any application.

Why it matters

One level of naming buys three things:

Upgrades in one place: move every feature to a new model version by editing one line.
Controlled rollouts: send 5 percent of an alias's traffic to a new model, compare, then increase.
Fast rollback: if the new model misbehaves, point the alias back.

In the real world: When a provider retires a model version, the platform team changes one alias and every feature moves over the same afternoon, after the test set has passed on the new version.

aliases:
  support-chat:
    - model: provider-a/medium-2025-06   # the pinned, exact version
      weight: 95
    - model: provider-a/medium-2025-11   # candidate, on 5% of traffic
      weight: 5
  cheap-fast:
    - model: provider-b/small-2025-03
      weight: 100

Letting an alias point at a provider's moving name such as 'latest'. The alias gives you control over when a model changes; pointing it at a name the provider controls hands that control straight back.

Practise it: Design an LLM Gateway

Virtual keys and authentication

What it is

The real provider keys are held only by the gateway. Each team or feature is given a virtual key of your own, which the gateway checks on every request and which carries:

who the caller is (team, feature, environment);
which model aliases it may use;
its rate limit and its budget.

Why it matters

Provider keys are powerful and expensive to leak: anyone holding one can spend your money. Keeping them in one service means a leaked virtual key can be cancelled in seconds without affecting anyone else, and every request is tied to a known caller, which is what makes limits and cost tracking possible at all.

In the real world: An intern commits a key to a public repository; because it was a virtual key scoped to a test project with a small budget, the damage is a cancelled key and nothing more.

Shipping a provider key inside a mobile app or a web page. Anything sent to a user's device can be extracted. Model calls must go from your servers, through the gateway, never directly from the user's device.

Rate limiting with a token bucket

What it is

A rate limit caps how fast a caller may send requests. The usual method is a token bucket:

each caller has a bucket that holds up to a fixed number of permits;
permits are added back at a steady rate;
each request takes a permit; with none left, the request is refused or queued.

For LLMs, limits are kept in both requests and model tokens per minute.

Why it matters

The bucket allows short bursts (up to its capacity) while holding the long-run rate to the refill rate.

That protects two things: the shared provider quota, so one runaway feature cannot use up everyone's allowance, and each team's budget.
Limits on model tokens matter more than limits on requests, because one huge request can cost as much as a thousand small ones.

In the real world: A batch job with a bug starts sending the same request in a loop; its own limit stops it within seconds, and the customer-facing chat on the same provider account never notices.

Visual example

Computed

A bucket of 10 permits, refilled 2 a second, against a burst. The burst is absorbed up to the capacity; the rest is refused.

Token bucket: capacity 10, refill 2 per second

second	requests arriving	allowed	refused	permits left
0	12	10	2	0
1	4	2	2	0
2	1	1	0	1
3	0	0	0	3
4	6	5	1	0
5	3	2	1	0

Over these 6 seconds, 20 of 26 requests got through.

import time

class TokenBucket:
    def __init__(self, capacity: float, refill_per_second: float):
        self.capacity, self.refill = capacity, refill_per_second
        self.level, self.updated = capacity, time.monotonic()

    def allow(self, cost: float = 1.0) -> bool:
        now = time.monotonic()
        self.level = min(self.capacity, self.level + (now - self.updated) * self.refill)
        self.updated = now
        if self.level < cost:
            return False            # refuse, or queue
        self.level -= cost
        return True

Counting only requests. Pass the estimated token count as the cost, and correct it with the real count when the response arrives. Otherwise a caller sending a few enormous prompts stays 'within limits' while using the whole quota.

Practise it: Design an LLM Gateway

Token metering and cost attribution

What it is

Metering is recording, for every request, the input tokens, output tokens, model and caller. Cost attribution is turning that into money per team, feature and customer:

tokens come from the provider's response, which reports the true counts;
cost is tokens times that model's prices;
each record is tagged with the caller, taken from its virtual key.

Why it matters

A single provider invoice says what the company spent, not who spent it or on what. With attribution, each team sees its own bill, an expensive feature is visible the week it launches, and a decision such as 'move order summaries to a smaller model' can be put in rupees.

In the real world: A monthly review shows one internal tool, used by twelve people, costing more than the customer chat used by a million; nobody had known, because all the spend sat under one shared key.

Visual example

Computed

One day's usage by feature, priced at the medium tier. The biggest spender is not the one most people would guess.

Daily cost by feature (₹250 in / ₹1,250 out per million tokens, example prices)

feature	input tokens	output tokens	cost per day
Support assistant	120M	12M	₹45,000
Order summaries	300M	20M	₹1 lakh
Product descriptions	40M	80M	₹1.1 lakh
Total			₹2.6 lakh

Product descriptions send the fewest input tokens and cost the most, because output tokens are five times the price of input.

Estimating tokens yourself and never reconciling. Your count will drift from the provider's, which is what you are charged for. Record the provider's reported usage, and handle streamed responses, where the count arrives only at the end.

Practise it: Design an LLM Gateway

Budget enforcement

What it is

A budget is a cap on spend for a team, feature or customer over a period. Budget enforcement is the gateway acting when the cap is near or reached:

at a warning level (say 80 percent), it alerts the team;
at the limit, it follows a chosen policy: refuse, slow down, or switch to a cheaper model.

Why it matters

LLM spend has no natural ceiling.

A bug that loops, a prompt that grew, or a sudden spike in traffic can multiply a bill overnight.
A budget turns an unbounded cost into a bounded one, and the choice of policy decides whether the product degrades gently or stops.

In the real world: A prompt change accidentally triples the tokens per request; the budget alert fires at lunchtime, and the team rolls back before the month's allowance is gone.

Visual example

Policies

Pick per feature. A customer chat and a nightly batch job deserve different answers.

What to do when a budget is reached

policy	what the user sees	suits
Alert only	Nothing changes	Early warning at 80 percent
Switch to a cheaper model	Slightly weaker answers	Customer-facing features
Slow down (queue)	Longer waits	Internal tools
Refuse	An error	Batch jobs, test environments

A hard stop on a customer-facing feature. Refusing every request when a budget runs out turns a cost problem into an outage. For user-facing features, prefer falling back to a cheaper model and paging a person; keep hard stops for internal and batch work.

Practise it: Design an LLM Gateway

The gateway as a critical service

What it is

Every model call now passes through the gateway, so it must be more reliable than anything behind it:

it keeps no state of its own in memory, so any copy can serve any request;
it runs as several copies behind a load balancer;
shared counters (limits, budgets) live in a fast shared store such as Redis;
it adds only a few milliseconds to each call.

Why it matters

A gateway that goes down takes every AI feature with it.

The design goal is that it fails open where it safely can: if the counter store is unreachable, serving requests with a local approximate limit is usually better than refusing everything.
Logging is done off the request path, so a slow log store never slows an answer.

In the real world: A logging database slows down one evening; because log writes go through a queue, answers keep flowing and the logs catch up an hour later.

Putting slow work on the request path. Writing full logs, running evaluations or calling an analytics service before returning the answer adds their delay and their failures to every request. Hand that work to a queue and return.