LLM System Design · Module 3 of 17 · 7 concepts · about 11 min

Choosing a Model

How to pick a model for a feature, whether to rent one through an API or run your own, and when prompting, retrieval or fine-tuning is the right way to get the behaviour you need.

At work: The first architecture decision on any AI feature, and one that is revisited every few months as new models appear and prices change.

What to compare when choosing a model

What it is

Six things separate one model from another for a given feature:

Quality on your task: measured on your own examples, not on a public leaderboard.
Speed: time to first token and tokens per second.
Price: per million input and output tokens.
Context window: how much it can read at once.
Features: tool calling, structured output, images, the languages you need.
Terms: where your data goes, whether it is kept, and what uptime is promised.

Why it matters

No model is best on all six.

The largest is the most capable and the slowest and most expensive.
The right choice is the cheapest, fastest model that is good enough for this feature, and 'good enough' can only be decided by testing on the feature's own examples.

In the real world: A travel company tests three models on a hundred real customer questions and picks the mid-sized one: it matched the largest on quality for this task at a fifth of the price.

Visual example

Checklist

The first row is the only one that needs your own data, and it matters most.

What to check before committing to a model

question	why it matters
How does it score on my own test examples?	The only number that predicts your results
How fast is the first token, and the rest?	Decides whether the feature feels responsive
What does a typical request cost?	Multiplied by every request, every day
Does my longest prompt fit?	Otherwise content must be cut or summarised
Tool calling, JSON output, images, my languages?	Missing features are hard to work around
Where does the data go and is it stored?	May rule a provider out entirely

Choosing by leaderboard rank or by whichever model is newest. Leaderboards measure general ability; your feature is one specific task, often in a specific language, with a specific budget. Twenty real examples tell you more than any ranking.

Model tiering

What it is

Providers offer models in sizes, and a system rarely needs the same size for everything. Model tiering means using:

a small model for simple, high-volume work (classifying, extracting, short replies);
a medium model for most everyday requests;
a large model only for the hard cases that need it.

Why it matters

The price gap between sizes is large.

With the example prices in this topic, the large tier costs 50 times what the small tier does per token. Most traffic in a real product is easy.
Sending it all to the largest model pays the top price for work a small model does as well, and makes every answer slower too.

In the real world: Kirana Kart uses a small model to write order summaries, a medium one for the support chat, and a large one only when the chat escalates a complicated refund dispute.

Visual example

Example tiers

The three example tiers used for every calculation in this topic. Each step up costs five to ten times more and runs at about half the speed.

Round illustrative figures, not a real price list

tier	₹ per million in	₹ per million out	first token (ms)	tokens per second	good for
small	25	125	300	150	Classifying, extracting, short replies
medium	250	1,250	600	80	Most everyday requests
large	1,250	6,250	1,200	40	Hard reasoning, long documents

Defaulting to the largest model 'to be safe' and never revisiting it. The safe default is the opposite: start from the smallest model that passes your tests and move up only where it fails.

Practise it: Design a ChatGPT-like Application

Hosted API or your own model

What it is

There are two ways to get a model:

Hosted API: you send requests to a provider and pay per token. Nothing to run.
Self-hosting: you run an open-weights model on your own (or rented) GPU servers and pay for the servers.

Why it matters

The choice is about four things:

Effort: an API works in an afternoon; self-hosting needs a team that can run GPUs.
Cost shape: an API costs nothing when idle; a server costs the same whether busy or empty.
Data control: self-hosting keeps every prompt inside your own systems.
Quality: the strongest models are usually available only through an API.

Most teams start with an API and move specific high-volume workloads to their own servers later, if the numbers justify it.

In the real world: A hospital group self-hosts a model because patient notes may not leave its network, accepting lower quality and a dedicated operations team as the price of that control.

Visual example

Comparison

Start with an API unless one of the right-hand reasons clearly applies.

Hosted API versus self-hosting

	hosted API	self-hosted
Time to first working version	Hours	Weeks
You pay for	Tokens used	Servers, busy or idle
Who runs it	The provider	Your team
Data leaves your systems?	Yes	No
Strongest models available?	Yes	Usually not
Choose it when	Starting out; variable traffic	Strict data rules; steady, very high volume

Self-hosting to save money without checking utilisation. A GPU that sits idle most of the day is more expensive per token than the API it replaced. The saving only appears when the server is kept busy.

The break-even point for self-hosting

What it is

Self-hosting pays off only above a certain level of use. To find it:

work out what the server costs per month;
work out how many tokens it could produce if it ran flat out;
price that many tokens at the API's rate;
the break-even utilisation is the server cost divided by that API cost.

Why it matters

The table works this through for one server at ₹250 an hour producing 2,000 tokens a second, against the small tier at ₹125 per million.

The server costs ₹1.8 lakh a month whatever happens.
It only beats the API when it is busy more than about 28 percent of the time, and that is before counting the engineers who keep it running.

In the real world: A startup works out that its overnight batch job would keep a GPU busy only four hours a day, and stays on the API; a year later, at ten times the volume, the same sum comes out the other way.

Visual example

Computed

One server at ₹1.8 lakh a month. It wins only when it is kept busy: break-even is about 28% utilisation.

Server: ₹250/hour, 2,000 tokens/second. API: ₹125 per million tokens. Illustrative.

how busy	tokens per month	API cost	server cost	cheaper
10%	0.5 billion	₹64,800	₹1.8 lakh	API
30%	1.6 billion	₹1.9 lakh	₹1.8 lakh	Own server
60%	3.1 billion	₹3.9 lakh	₹1.8 lakh	Own server
100%	5.2 billion	₹6.5 lakh	₹1.8 lakh	Own server

Break-even

server: ₹250 × 720 hours = ₹1.8 lakh a month

flat out: 2,000 × 3,600 × 720 = 5.2 billion tokens, worth ₹6.5 lakh at the API

break-even: ₹1.8 lakh ÷ ₹6.5 lakh = 28% utilisation

Comparing a server's cost at full use with the API's cost at your real use. Real traffic has quiet nights and busy evenings. Use your actual average utilisation, and add the cost of the people.

Prompting, retrieval or fine-tuning

What it is

Three ways to make a model behave the way a feature needs, from cheapest to most effort:

Prompting: write better instructions and examples. Changes take minutes.
Retrieval (RAG): look up the relevant facts and put them in the prompt. For knowledge the model lacks.
Fine-tuning: train the model further on your own examples. For a style, a format or a narrow skill that prompting cannot reach.

Why it matters

They fix different problems, so the order matters:

If the model does not know a fact, retrieval is the fix. Fine-tuning is a poor way to add facts.
If it knows what to do but does it inconsistently, improve the prompt first.
If the prompt has grown huge to force a behaviour, or a small model must match a large one on one task, consider fine-tuning.

Fine-tuning comes last because it needs data, training and re-doing whenever the base model changes.

In the real world: A support team first rewrites its prompt, then adds retrieval over the help centre, and only fine-tunes a small model a year later, to cut the cost of a task the large model already did well.

Visual example

Decision

Try them in this order.

Which tool for which problem

the problem	reach for	why
Answers are vague or badly formatted	Prompting	Clearer instructions and examples fix most of it
The model lacks your facts, or they change	Retrieval (RAG)	Facts live in documents, not in the model
Needs a consistent house style or format	Prompting, then fine-tuning	Style is learned well from examples
A small model must do one task very well	Fine-tuning	Buys large-model quality at small-model cost
Counting or totals over data	A database query	Not a language task at all

Fine-tuning as the first move. Teams spend weeks collecting data to fix what a clearer prompt or a retrieved document would have fixed in an afternoon.

Model versions and deprecation

What it is

Models are not permanent. Providers:

release new versions that behave slightly differently;
move a general name (such as 'latest') to a newer model without warning you;
retire old versions on a published date, after which calls to them fail.

Why it matters

A prompt tuned on one version can behave differently on the next: a format that was reliable becomes unreliable, a refusal appears, costs change. So a production system pins an exact version, keeps a test set to run against any new version before switching, and tracks retirement dates like any other dependency.

In the real world: A company's document extractor starts returning malformed output on a Tuesday with no deployment; the model name it used had been moved to a new version that morning.

Pointing production at a name that moves, such as 'latest'. The behaviour of your product then changes on the provider's schedule, with no release on your side and nothing in your logs to explain it.

Avoiding lock-in to one provider

What it is

Lock-in means your system only works with one provider's models. It builds up through:

prompts tuned to one model's quirks;
code written against one provider's request format;
features only that provider offers.

Why it matters

Being able to switch is worth real money and reliability:

when the provider has an outage, you can fail over;
when a cheaper or better model appears, you can move;
when prices or terms change, you have a choice.

The protection is a thin layer of your own between the application and any provider, which is the gateway in the next module, plus a test set that tells you whether a different model is good enough.

In the real world: During a provider's three-hour outage, one company's assistant keeps working on a second provider's model, while a competitor's shows an error page for the whole afternoon.

Assuming models are interchangeable because the request format is the same. The same prompt can produce noticeably different quality on another model. Switching needs testing, not only a changed setting.