LLM System Design · Module 5 of 17 · 7 concepts · about 10 min

Model Routing

Deciding, per request, which model should answer it. Done well, it cuts cost and waiting time sharply with little loss of quality.

At work: The biggest single cost lever a platform team has: most requests are easy, and routing stops the company paying large-model prices for them.

Why route at all

What it is

Routing means choosing a model per request, not one model for everything. A router can decide by:

task: which feature is calling;
difficulty: how hard this particular request looks;
cost or speed: the caller's budget or deadline;
availability: which providers are healthy right now.

Why it matters

Real traffic is uneven.

In most products a large share of requests are simple: a greeting, a status question, a one-line lookup.
With the example tiers in this topic, a request costs ₹0.11 on the small model and ₹5.63 on the large one, and takes 2.3 seconds against 8.7.
Routing the easy share to the small model saves most of the bill and makes those answers faster as well.

In the real world: A telecom's assistant finds that sixty percent of its chats are 'what is my balance' and 'when does my plan expire', and moves exactly those to a small model.

Adding routing before measuring the traffic. If nearly every request really is hard, a router adds cost and complexity and saves nothing. Label a sample of real requests as easy or hard first.

Static routing by task

What it is

The simplest router is a table: each feature, through its model alias, is assigned a model that suits it.

Order summaries: a small model.
Support chat: a medium model.
Contract analysis: a large model.

The decision is made once, by people, from test results. Nothing is decided per request.

Why it matters

It captures most of the benefit for none of the complexity.

It is predictable, easy to test and easy to explain.
Start here, and add per-request routing only inside a feature whose requests vary widely in difficulty.

In the real world: An audit of an insurer's AI features finds eleven of fourteen running on the largest model by default; moving eight of them to smaller tiers halves the bill with no measurable quality change.

Skipping this step and building a clever per-request router first. If the features themselves are on the wrong tier, the cleverest router is tuning the wrong thing.

Routing by difficulty

What it is

Within one feature, a router looks at each request and predicts whether a small model can handle it. The router itself can be:

rules: length, keywords, whether an order number is present;
a small classifier: a cheap model trained on labelled examples of easy and hard requests;
a small LLM: asked to label the request, at the cost of one short call.

Why it matters

The router must be much cheaper and faster than the model it is choosing between, or it eats its own saving.

It also makes mistakes in two directions: sending a hard request to the small model lowers quality, and sending an easy one to the large model wastes money.
The first kind is the one users notice, so routers are tuned to be cautious.

In the real world: A coding assistant sends short completions to a small model and multi-file refactoring requests to a large one, using a classifier that runs in under ten milliseconds.

def route(request: str) -> str:
    if len(request) < 200 and looks_like_lookup(request):
        return "small"                      # cheap rule first
    label = classifier.predict(request)      # 'easy' or 'hard', in a few ms
    return "small" if label == "easy" else "large"

Judging a router by its accuracy alone. What matters is the final answer quality and cost across all traffic. A router that is 90 percent accurate but sends the costliest mistakes the wrong way can be worse than no router.

Cascades: try the small model first

What it is

A cascade does not predict difficulty in advance. It:

sends every request to the small model first;
checks whether the answer is good enough;
escalates to a larger model only when it is not.

The check can be the model's own stated confidence, a validation rule (did it return valid JSON, did it cite a source?), or a quick judge.

Why it matters

A cascade needs no trained router and adapts to each request by its result.

Its weakness is the hard request: that one pays for the small model, waits for it, and then pays and waits for the large one as well.
So a cascade works best when most requests are easy and the check is cheap and reliable.

In the real world: A document-extraction service runs a small model first and escalates only when the extracted totals do not add up, which happens on about one invoice in five.

Visual example

Cascade

Easy requests stop at step 2. Hard ones go on and pay for both models.

A two-step cascade

Small model answersFast and cheap, every request
Check the answerFormat, source, rules
Escalate if it failsSame request to the large model
ReturnFrom whichever model passed

Trusting the small model's own confidence. Models are often confident when wrong. Prefer checks that can be verified: the output parses, the cited passage exists, a required field is present.

What routing saves

What it is

The saving depends on two numbers:

the share of easy requests;
the price gap between the models.

The table works it through for a request of 3,000 input and 300 output tokens, with 70 percent of traffic easy.

Why it matters

Putting numbers on it is what turns routing from a nice idea into a decision.

It also shows the difference between the two approaches: a router adds a small fixed cost to every request; a cascade adds a large cost to the hard ones.
Which is cheaper depends on the share of easy traffic and on how good the router is.

In the real world: A platform team presents routing to finance as one line: 'same quality on our test set, sixty-odd percent off the model bill', with both numbers measured.

Visual example

Computed

Four strategies for the same traffic, 70% of it easy. Routing or cascading saves about two thirds against sending everything to the large model.

Per request: 3,000 tokens in, 300 out. Example prices. 1,000,000 requests a month.

strategy	₹ per request	per month	saving	catch
Everything to the large model	5.63	₹56.3 lakh	0%	Simplest; pays top price for easy work
Router: easy to small, hard to large	1.78	₹17.8 lakh	68%	One extra small call per request
Cascade: small first, escalate if unsure	1.8	₹18 lakh	68%	Hard requests pay for both models
Everything to the small model	0.11	₹1.1 lakh	98%	Cheapest; fails the hard requests

How each is priced

small: ₹0.1125 per request

large: ₹5.625 per request

router: router call ₹0.0106 + 0.7 × small + 0.3 × large = ₹1.78

cascade: small for all + 0.3 × large = ₹1.8

Reporting the saving without the quality. A table like this is only half the argument. Next to it must sit the test-set score for each strategy, or you have just measured how to spend less on worse answers.

Load balancing across providers and regions

What it is

The same model is often reachable through more than one route: several regions, several accounts, or more than one provider. The gateway spreads traffic across them by:

weights: fixed shares, such as 70 and 30;
least busy: whichever route has the most rate-limit headroom;
fastest: whichever has had the lowest recent response time;
health: skipping routes that are failing.

Why it matters

Each route has its own tokens-per-minute limit, so several routes together give more total capacity than any one.

Spreading also means one region's slowdown affects only a share of traffic.
This is the same idea as a load balancer in front of web servers, except that what is being balanced is quota and latency, not processor load.

In the real world: A company doubles its usable capacity at peak by splitting traffic between two regions of the same provider, each with its own per-minute allowance.

Balancing in a way that defeats prompt caching. Caches are kept per route. Spraying requests that share a long common prefix across many routes means each one sees a cold cache. Keep related traffic on the same route where caching matters.

Testing a new model with routing

What it is

Routing is also how a new model or prompt is introduced safely:

Shadow: the new model receives a copy of real requests; its answers are recorded, never shown.
Canary: a small share of real users, say 5 percent, get the new model.
A/B test: two versions run side by side on random halves, and outcomes are compared.

Why it matters

A test set tells you how a model does on yesterday's questions.

Real traffic contains tomorrow's.
Shadowing shows the new model's behaviour on live requests at no risk to users; a canary limits the damage of a bad change to a small share; an A/B test measures whether the change improved what the business cares about.

In the real world: A bank runs a new model in shadow for two weeks, finds it refuses a class of perfectly reasonable questions, and fixes the prompt before any customer sees it.

Visual example

Rollout

In increasing order of risk and of what you learn.

Three ways to try a new model on real traffic

method	who sees the new answers	risk to users	tells you
Shadow	Nobody	None	How it behaves on real requests
Canary	A small share, such as 5%	Small	Whether anything breaks in production
A/B test	A random half	Moderate	Whether it is actually better

Reading a canary's cost and speed and calling it a success. Those are easy to measure. Whether answers got better needs an evaluation of quality on the same traffic, which is the harder half and the one that matters.