LLM System Design · Module 11 of 17 · 8 concepts · about 11 min

Cost Control

How to work out what a feature will cost before building it, see where the money actually goes, and reduce it without hurting quality.

At work: The conversation every AI team has with finance: what does this cost per user, why is the bill rising, and what can be cut.

The cost model

What it is

The monthly cost of a feature is four numbers multiplied together:

requests per month;
tokens per request, input and output counted separately;
the price per token of the model used;
plus smaller amounts for embeddings, reranking and extra model calls.

cost = requests × (input tokens × input price + output tokens × output price).

Why it matters

Every way of cutting cost acts on one of those numbers.

Caching lowers requests. Trimming context lowers input tokens. A length cap lowers output tokens. A smaller model lowers the price.
Knowing the formula tells you which lever to pull, and lets you estimate a feature's bill on a whiteboard before any code exists.

In the real world: Before approving a new feature, a product lead asks for one number, cost per active user per month, and the engineer derives it in two minutes from this formula.

Support assistant, medium tier (₹250 in / ₹1,250 out per million):

  per request   3,700 in + 300 out          = ₹1.3
  per month     ₹1.3 x 1,000,000 requests  = ₹13 lakh

Forgetting the hidden calls. A request that first rewrites the question, then classifies it, then answers, then checks the answer, is four model calls. Count every call the request makes, not only the one that produces the visible answer.

Where the money goes

What it is

Break one request down by the part of the prompt that each token came from:

instructions;
user details;
retrieved documents;
conversation history;
the answer.

Why it matters

The breakdown almost always surprises people.

The visible answer is a small share.
Retrieved documents and conversation history are usually the largest, and both can be cut by sending less.
Without this view, teams try to save money by shortening the instructions, which are a rounding error.

In the real world: A breakdown shows a team that 60 percent of its spend is retrieved text; cutting from eight passages to four, after checking quality, removes a quarter of the bill.

Visual example

Computed

One support request, ₹1.3, split by part. Documents and history are most of it; the instructions are not.

Medium tier, example prices. 3,700 tokens in, 300 out.

part	tokens	₹	share of cost
Instructions (system prompt)	500	0.125	10%
Customer and order details	300	0.075	6%
Retrieved policy passages	1600	0.4	31%
Conversation history	1200	0.3	23%
The new question	100	0.025	2%
The answer (output)	300	0.375	29%
Total	4000	1.3	100%

Optimising the part you wrote. The system prompt is the part engineers see every day, so it gets the attention, while the documents and history that are assembled automatically, and cost five times as much, go unexamined.

Reducing input tokens

What it is

Input is trimmed by sending the model less to read:

fewer, better passages: retrieve widely, rerank, keep only the best few;
shorter history: a window or a summary in place of the whole chat;
compression: keep only the sentences that bear on the question;
a leaner prompt: remove instructions and examples that no longer earn their place;
prompt caching: pay a fraction for the part that never changes.

Why it matters

Input tokens are the bulk of most bills, and they are also where quality and cost pull the same way: a focused prompt is usually both cheaper and more accurate than a stuffed one. That makes input the first place to look.

In the real world: A legal assistant halves its cost by extracting the relevant sentences from each long clause before sending them, and its answers become more precise.

Cutting context without re-running the test set. Halving the passages may be free or may remove the one that held the answer. The only way to know is to measure quality before and after.

Reducing output tokens

What it is

Output is trimmed by asking the model to write less:

a maximum output setting as a hard cap;
an instruction for the length wanted ('two sentences', 'three bullet points');
structured output in place of prose, where a program is the reader;
no restating of the question, no closing pleasantries.

Why it matters

Output tokens cost several times what input tokens do (5 times, on the example tiers) and are also what the user waits for. A shorter answer is cheaper and faster at once, which makes it the rare change with no trade-off, provided the answer still does its job.

In the real world: An order-summary feature cuts its output from a paragraph to a single line of structured fields; the delivery partners prefer it, and it costs a fifth as much.

Relying on the cap alone. A cap cuts the answer off mid-sentence when it is reached. Ask for a short answer in the prompt, and keep the cap as a safety net well above the length you asked for.

Batch processing for offline work

What it is

Many providers offer a batch service for work nobody is waiting on:

you submit a large file of requests;
results come back within hours, not seconds;
the price is discounted, commonly by about half.

Why it matters

A large share of LLM work is not live at all: generating product descriptions, classifying yesterday's tickets, building evaluation runs, summarising documents overnight.

Paying live prices for it buys speed that nobody uses.
For the running example's product descriptions, the batch price turns ₹13,000 a month into ₹6,500.

In the real world: A marketplace moves its nightly listing clean-up to a batch service; the job now finishes by morning at half the price, and the evening chat no longer hits rate limits.

Running offline jobs through the live path because it already exists. They compete with customer traffic for the same rate limit, at full price, and slow the chat down at peak.

Forecasting a feature's cost

What it is

A forecast lists each feature with its own numbers:

requests per month;
typical input and output tokens;
the model tier it uses;
any discount (batch, caching).

Multiply through, and add them up.

Why it matters

Doing it per feature shows what a single total hides: which feature dominates, and which lever would matter. It also gives a baseline, so that when the real bill arrives higher, you can see which of the four numbers moved: more requests, longer prompts, longer answers or a dearer model.

In the real world: A forecast made before launch shows order summaries would cost more than the chat if run on the same model; they are moved to a small tier before a single request is served.

Visual example

Computed

The three running-example features for one month. Total ₹14.7 lakh, most of it one feature.

Monthly forecast by feature (example prices)

feature	tier	requests	tokens in / out	per request ₹	per month
Support assistant	medium	1,000,000	3,700 / 300	1.3	₹13 lakh
Order summaries	small	6,000,000	800 / 60	0.028	₹1.7 lakh
Product descriptions (batch)	small	200,000	600 / 400	0.033	₹6,500
Total					₹14.7 lakh

The batch row includes a 50% discount. Six times the requests on the small tier still cost far less than the chat on the medium one.

Forecasting with average token counts and no margin. Real traffic has long conversations and big documents in its tail, and they cost many times the average. Check the 95th percentile request, and add headroom.

Cost per outcome, not per call

What it is

The number that matters to a business is the cost of the result, not of one model call:

cost per resolved support conversation;
cost per document processed correctly;
cost per active user per month.

It includes every call in the conversation, and what happens when the model fails and a person takes over.

Why it matters

A cheaper model that resolves fewer conversations can cost more overall, because each failure is handed to a human.

In the worked example, a 6-turn chat costs ₹7.8 in model calls.
If 60 percent are resolved without a person and each handover costs ₹40, the true average is ₹23.8 per conversation.
Raising the resolved share is worth far more than shaving the model price.

In the real world: A support leader rejects a move to a cheaper model after a test shows it would save two rupees per chat in model cost and add six in agent time.

Visual example

Computed

What one support conversation really costs when 40% are handed to a person. The handovers, not the model, are most of the cost.

6 turns per chat at ₹1.3 a turn; a human handover costs ₹40

item	working	₹
Model calls per conversation	6 × ₹1.3	7.8
Human handovers, averaged	0.4 × ₹40	16
True cost per conversation	sum of the two	23.8
Model cost per resolved conversation	₹7.8 ÷ 0.6	13
A person handling it from the start		40

Choosing the model with the lowest price per token. If it resolves ten percent fewer conversations, the extra human handovers cost more than the model saved. Compare models on cost per successful outcome.

The order to pull the cost levers

What it is

Roughly from most to least effective for the effort:

Right-size the model per feature, and route easy requests to a small one.
Send less input: fewer passages, shorter history.
Cap and shorten output.
Cache: prompt caching everywhere, response caching where requests repeat.
Batch anything nobody is waiting for.
Fine-tune or self-host only at high, steady volume.

Why it matters

The first three are configuration changes that can be made in a day and often remove more than half the bill.

The last is a project of months.
Teams that start at the bottom of the list spend a quarter building what a week at the top would have made unnecessary.

In the real world: A company cuts its model bill by seventy percent in a month using only the first four levers, with its test-set score unchanged, and shelves its plan to self-host.

Visual example

Levers

Each acts on one term of the cost formula.

Cost levers, by which number they change

lever	reduces	effort	risk to quality
Smaller model / routing	Price per token	Low	Check on the test set
Fewer passages, shorter history	Input tokens	Low	Check on the test set
Shorter answers, output cap	Output tokens	Low	Low
Prompt caching	Price of repeated input	Low	None
Response caching	Number of calls	Medium	Stale or wrong hits
Batch service	Price per token	Low	None (but slow)
Fine-tune a small model / self-host	Price per token	High	Needs its own evaluation

Cutting cost without watching quality. Every lever here can make answers worse. Each change is made behind the test set, and the saving is reported together with the quality score, never alone.