LLM System Design · Module 2 of 17 · 7 concepts · about 12 min

How a Model Call Works

The handful of facts about one model call that decide everything else: tokens, the context window, how a call is priced, where its time goes, and why the model has no memory.

At work: The numbers an engineer reaches for when estimating what a feature will cost and how fast it will feel, before any code is written.

Tokens

What it is

A model does not read letters or words. It reads tokens: pieces of text, each a short word or a fragment of a longer one.

In English, 1 token is about three quarters of a word, so 100 tokens is roughly 75 words.
Other languages, code and unusual words use more tokens for the same meaning.
Everything is counted in tokens: the prompt you send, the answer you get, the limits and the bill.

Why it matters

Tokens are the unit of the whole subject.

Cost is tokens times a price. Speed is tokens per second. Limits are tokens per minute. A context window is a number of tokens.
If you can estimate the tokens in a request, you can estimate its cost, its speed and whether it fits.

In the real world: A team budgeting a multilingual assistant finds that the same question costs noticeably more in Tamil than in English, because the text splits into more tokens.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("Where is my refund for order 88213?")
len(tokens)        # about 10 tokens for 7 words

# A rough rule when you cannot count exactly:
#   tokens ~ words / 0.75        (English)
#   tokens ~ characters / 4

Estimating in words or characters and forgetting the conversion. A 3,000-word document is about 4,000 tokens, not 3,000, and in Hindi or in code the gap is larger. Different models also count the same text slightly differently.

The context window

What it is

The context window is the largest number of tokens a model can handle in one call, counting both what you send and what it writes.

It holds your instructions, the conversation so far, any documents you include and the answer.
Sizes range from a few thousand tokens to a million, depending on the model.
Anything that does not fit must be cut, summarised or left out.

Why it matters

The window is a hard limit and also a soft one.

Past the hard limit the call is rejected.
Well before it, a fuller window costs more, runs slower, and makes the model more likely to miss a detail in the middle.
So the real question is not 'does it fit?' but 'what deserves to be in there?'.

In the real world: A legal assistant that pasted entire 200-page contracts into each prompt cut its bill by ninety percent by sending only the relevant clauses, and its answers got more accurate.

Treating a large window as free space. A model with a million-token window still charges for every token on every call, and its accuracy on a fact buried in a very long prompt is lower than on the same fact in a short one.

Practise it: Design a ChatGPT-like Application

Input and output tokens are priced differently

What it is

Providers charge per token, with two prices:

an input price for the tokens you send (the prompt);
an output price for the tokens the model writes, usually several times higher.

The cost of one call is input tokens × input price + output tokens × output price.

Why it matters

Two things follow, and they shape most cost decisions:

Input is cheap per token and large in volume, so long prompts quietly dominate the bill.
Output is expensive per token, so capping answer length is the cheapest saving there is.
Model size multiplies both: the same call on a large model can cost fifty times what it costs on a small one.

In the real world: A finance team asks why a chatbot that writes two-sentence answers costs so much; the breakdown shows that each two-sentence answer is preceded by four pages of prompt.

Visual example

Computed

The same call (3,000 tokens in, 300 out) on three model sizes. Model choice moves the bill more than anything else.

Cost of one support reply, by model tier (round example prices, ₹ per million tokens)

tier	price in / out	input ₹	output ₹	per call ₹	per million calls
small	₹25 / ₹125	0.075	0.038	0.11	₹1.1 lakh
medium	₹250 / ₹1,250	0.75	0.375	1.13	₹11.3 lakh
large	₹1,250 / ₹6,250	3.75	1.875	5.63	₹56.3 lakh

The prices are round illustrative figures. Use your provider's current prices; the method is the same.

One support reply: 3,000 input tokens, 300 output tokens, on the medium tier (₹250 in, ₹1,250 out, per million tokens).

input: 3,000 ÷ 1,000,000 × 250 = ₹0.75
output: 300 ÷ 1,000,000 × 1,250 = ₹0.375
one call: ₹1.13
a million calls a month: ₹11.3 lakh

Estimating cost from the visible answer. The answer is a few hundred tokens. The prompt behind it, with instructions, history and documents, is several thousand. Forgetting the input understates the bill badly.

Where the time goes: time to first token and tokens per second

What it is

A model writes its answer one token at a time, so a call has two parts:

Time to first token (TTFT): the wait before anything appears, while the model reads the prompt.
Generation: the answer arriving token by token, at some number of tokens per second.

Total time is roughly TTFT + output tokens ÷ tokens per second.

Why it matters

This formula explains most latency decisions:

Output length dominates. A 1,000-token answer takes many times longer than a 50-token one.
Input length matters far less: reading is much faster than writing.
With streaming, the user starts reading at TTFT, so TTFT is what they feel.
Smaller models are faster on both parts.

In the real world: A summarisation feature drops from eleven seconds to four when the team asks for 'three bullet points' in place of 'a detailed summary'; the prompt was not changed at all.

Visual example

Computed

One model (first token after 600 ms, then 80 tokens a second) writing answers of three lengths. The answer's length decides the wait.

total ms = time to first token + output tokens ÷ tokens per second

output tokens	first token (ms)	writing (ms)	total (ms)
50	600	625	1225
300	600	3750	4350
1000	600	12500	13100

Speeds are illustrative. Measure your own model: both numbers vary with the provider, the model and the time of day.

Trying to speed up a slow feature by shortening the prompt. Unless the prompt is enormous, the time is in the answer. Ask for a shorter answer, or stream it.

The model has no memory

What it is

A model keeps nothing between calls. Each call is answered from the text in that call alone.

A chat only appears to remember, because the application sends the whole conversation again every turn.
Turn 10 therefore includes turns 1 to 9 as input.
The input grows every turn, and so does the cost of each turn.

Why it matters

This one fact explains why long chats get slower and more expensive, why history has to be trimmed or summarised, and why 'memory' is something you build in your own storage, not something the model has. It is also why any server can handle any request: the state lives with you.

In the real world: A support team notices that long chats cost ten times more per message than short ones, and adds a rule to summarise the conversation after fifteen turns.

Visual example

Computed

Input tokens sent on each turn of one chat: 550 on turn 1, 6,250 on turn 20. The line climbs because every turn resends all the turns before it.

Per turn the input grows in a straight line; added up over the chat, the total (68,000 tokens) grows with the square of the number of turns.

A chat with a 500-token system prompt, 50-token questions and 250-token answers, resending everything each turn:

turn 1: 550 input tokens
turn 2: 850 input tokens
turn 5: 1,750 input tokens
turn 10: 3,250 input tokens
turn 20: 6,250 input tokens

Over 20 turns the application sends 68,000 input tokens in total, though the user only typed 1,000.

Assuming the provider stores the conversation for you. Unless you use a feature built for that, you must store the messages and send them back each time, and you pay for them each time.

Generation settings

What it is

A few settings on each call control how the answer is produced:

Temperature: how much randomness. Low gives the most likely wording every time; high gives more varied text.
Maximum output tokens: a hard cap on the answer's length.
Stop sequences: text that ends the answer early when it appears.
Top-p: another randomness control, which limits the choice to the most likely tokens.

Why it matters

They are small levers with large effects:

Low temperature suits facts, extraction and anything that must be consistent.
Higher temperature suits brainstorming and creative writing.
The output cap bounds cost and time, and is the first safety net against runaway answers.

In the real world: An invoice-extraction feature runs at temperature zero with a tight output cap, while the same company's marketing-copy tool runs warm and long.

Visual example

Settings

Defaults to start from, then adjust by measuring.

The settings and their usual values

setting	what it controls	factual tasks	creative tasks
Temperature	Randomness of wording	0 to 0.3	0.7 to 1
Max output tokens	Longest possible answer	Tight cap	Generous cap
Stop sequences	Where to end early	Often used	Rarely

Setting temperature to zero and expecting identical answers every time. It makes answers far more consistent, not guaranteed identical. Never build logic that depends on an exact repeat.

Rate limits: requests and tokens per minute

What it is

A provider limits how much each customer can use, in two ways at once:

Requests per minute (RPM): how many calls.
Tokens per minute (TPM): how many tokens across all calls.

Go over either and the provider rejects the call with a 'too many requests' error (HTTP status 429).

Why it matters

For an LLM system the token limit is the real capacity limit.

A few very long requests can use up a minute's allowance as surely as thousands of short ones.
Capacity planning is therefore done in tokens per minute, and a system must expect to be refused and know what to do next: wait and retry, queue, or switch to another model.

In the real world: A retailer's assistant fails every evening at eight o'clock; the cause is not a bug but the tokens-per-minute limit, reached when order-tracking questions peak.

Limit: 2,000,000 tokens per minute

Each support request:  3,000 in + 300 out = 3,300 tokens
Requests per minute:   2,000,000 / 3,300   = about 600
Requests per second:   600 / 60            = about 10

At 25 requests a second at dinner time, this limit is too small.

Planning capacity in requests per second, as for a web server. Two features with the same request rate can differ twenty times in tokens, and it is tokens that run out.