ML System Design · Module 19 of 47 · 11 concepts · about 13 min

Inference Optimisation and Scaling

Making predictions fast and cheap at scale: measuring where time goes, batching requests, shrinking models with quantisation, distillation and pruning, choosing CPUs or GPUs, autoscaling, sharing hardware between models, and taming the slowest requests.

At work: What turns an accurate model that is too slow or too expensive into one that can ship. Serving cost often exceeds training cost within weeks of launch.

Measure where the time goes

What it is

Before optimising, measure each step of a prediction at p50 and p99:

network into and out of the service;
feature fetching;
pre-processing (building the input row);
the model itself;
post-processing and logging.

Then optimise the largest step first.

Why it matters

The model is often not the slowest part.

Feature lookups made one at a time, slow JSON parsing or synchronous logging frequently cost more than the model.
Speeding up the model by half saves nothing if it was 10% of the time.

In the real world: Profiling Kirana Kart's ETA service shows the model takes 2 ms of a 25 ms request; 15 ms is spent on four separate feature-store calls, which one batched call reduces to 4 ms.

Quantising or distilling a model before profiling. A week of model work saves 3 ms; batching feature reads would have saved 20.

Batching requests

What it is

Each model call has a fixed overhead (launching work on the hardware, moving data) plus a cost per item. Scoring many items in one call spreads the overhead:

within a request: score all 500 candidates in one call, never one at a time;
across requests (dynamic batching): the server waits a few milliseconds to group requests from different users into one call.

Model servers such as Triton do dynamic batching with two settings: the largest batch and the longest wait.

Why it matters

Batching can multiply throughput, especially on GPUs, whose overhead is high and per-item cost low.

Throughput rises with batch size until the per-item cost dominates.
Latency rises too, because of the wait and the bigger call; the maximum wait is a direct latency trade.

In the real world: GPU-served recommendation and vision models almost always use dynamic batching; a wait of a few milliseconds is a common setting.

Visual example

Computed

One GPU call costs 5 ms plus 0.2 ms per item. Bigger batches serve far more items a second, each call taking longer.

call time = 5 + 0.2 × batch size

batch size	call time	items a second	vs batch of 1
1	5.2 ms	192	1×
8	6.6 ms	1,212	6.3×
32	11.4 ms	2,807	14.6×
128	30.6 ms	4,183	21.8×

Setting the maximum wait higher than the latency budget allows. Throughput looks wonderful in a load test; every real request is 20 ms slower.

Quantisation

What it is

Quantisation stores a model's numbers with fewer bits:

from 32-bit floating point to 16-bit (half the size; little or no accuracy loss);
to 8-bit integers (a quarter of the size; usually a small loss, checked on validation data);
lower still for very large models, with more care.

Either after training (post-training quantisation, with a small calibration set) or during it (quantisation-aware training, for less loss).

Why it matters

Smaller numbers mean less memory and faster arithmetic.

Modern CPUs and GPUs have fast paths for 16- and 8-bit maths.
Smaller models fit on cheaper hardware and on phones.
The accuracy cost must be measured on the same evaluation set and slices as any other change.

In the real world: Kirana Kart's on-phone search-suggestion model is quantised to 8 bits, which takes it from 40 MB to 10 MB with no measurable change in suggestion quality.

Visual example

Computed

A 100,000,000-parameter model at each precision.

Size = parameters × bytes per parameter

precision	bytes each	model size
32-bit float	4	400 MB
16-bit float	2	200 MB
8-bit integer	1	100 MB

Quantising and shipping without re-running the evaluation. Some layers or rare slices lose much more accuracy than the average suggests.

Practise it: Design an Image Search System

Distillation

What it is

Knowledge distillation trains a small student model to imitate a large teacher:

run the teacher over lots of data and record its predictions (its full probabilities, not just its top answer);
train the student to match those predictions;
serve the student.

Why it matters

The student often keeps most of the teacher's accuracy at a fraction of the cost.

The teacher's soft probabilities carry more information than hard labels ('70% this, 25% that').
Unlabelled data becomes useful, since the teacher labels it.
A common pattern: a heavy model offline, a distilled one online.

In the real world: DistilBERT kept most of BERT's language-understanding accuracy with about 40% fewer parameters and was roughly 60% faster, which made it a common choice for production text models.

Distilling into a student too small to learn the task. If the student's validation accuracy plateaus far below the teacher's, it needs more capacity or better features, not more data.

Smaller models by design

What it is

Other ways to make a model cheaper:

pruning: remove weights or neurons that contribute little;
fewer trees, shallower trees for boosted models;
fewer features: each one removed saves fetch and compute time (module 9);
compiling the model for the target hardware (ONNX Runtime, TensorRT, OpenVINO);
smaller architectures designed for speed.

Why it matters

Often the cheapest change is simply a smaller version of the same model.

A boosted model with 2,000 trees may score almost identically with 400.
Compilers can speed up a model severalfold with no change to its accuracy.

In the real world: Kirana Kart cut its fraud model from 1,500 to 500 trees after seeing validation PR-AUC flatten at 450, tripling its speed.

Assuming more trees or layers always help. Plot validation accuracy against size; the curve usually flattens long before the default settings.

CPU or GPU

What it is

GPUs are much faster for large neural networks and big batches. CPUs are cheaper per hour and fine for:

boosted trees and linear models;
small neural networks;
low or spiky traffic.

Compare cost per thousand predictions at your real traffic, including the minimum number of replicas you must run for reliability.

Why it matters

Cheaper per prediction does not mean cheaper overall.

A GPU is efficient only when kept busy.
At modest traffic, the minimum of two replicas (so one can fail) leaves expensive GPUs mostly idle.
At high traffic, the GPU's efficiency wins.

In the real world: At its current traffic Kirana Kart serves the ranker on CPUs; the analysis here shows GPUs would only pay off at several times today's peak.

Visual example

Computed

The ranking service at Kirana Kart's peak of 278 requests a second, with 30% headroom and at least 2 replicas. The GPU is cheaper per request, yet dearer per month at this traffic.

Illustrative throughput and prices

option	requests a second each	₹ per hour	₹ per 1,000 at full use	replicas needed	₹ a month
CPU replica	50	40	0.222	8	₹2.3 lakh
GPU replica	600	250	0.116	2	₹3.6 lakh

Choosing GPUs because the model is 'deep learning'. Many small neural models serve perfectly well on CPUs, especially after export to ONNX.

Autoscaling

What it is

Autoscaling adds replicas when load rises and removes them when it falls. Choose the signal carefully:

CPU or GPU use: simple, but lags behind sudden spikes;
requests a second or queue depth: reacts to load directly; queue depth suits batch and asynchronous work;
latency: scale when p99 approaches the budget.

Set a minimum (for reliability and the first requests of a spike) and a maximum (for cost).

Why it matters

Traffic varies by hour and day; paying for peak capacity all night wastes money.

Starting a replica takes time (loading the model, warming up), so scaling must start before the service is overwhelmed.
Scale to zero suits rarely used models, accepting a slow first request (cold start).

In the real world: Kirana Kart runs its ranker at a minimum of four replicas overnight and lets it scale to twelve in the evening rush, triggered by requests per replica.

Scaling on CPU use for a GPU model, or ignoring start-up time. A replica that takes two minutes to load a large model arrives after the spike has already caused timeouts.

Practise it: Design a Model Serving Platform

Many models on shared hardware

What it is

Companies run dozens or hundreds of models. Ways to share hardware efficiently:

multi-model servers load several models in one process (Triton, KServe ModelMesh);
GPU sharing: several models on one GPU, by time-slicing or partitioning (NVIDIA MIG);
resource limits and isolation so one model's spike cannot starve the others;
priorities: live models before batch jobs.

Why it matters

One replica pair per model is wasteful when most models get little traffic.

Sharing can cut hardware by a large factor.
Without isolation, a busy or leaking model degrades every model next to it.

In the real world: Kirana Kart packs fifteen low-traffic models (category tagging, image quality, address cleaning) onto one shared Triton pool, while the ranker and ETA models have dedicated replicas.

Putting a latency-critical model on the same GPU as a batch job with no priority. The batch job fills the GPU and the live model misses its budget.

Taming tail latency

What it is

The slowest requests (p99, p99.9) are what users feel and what breaks budgets. Causes and cures:

cold starts and lazy loading: warm up models with dummy requests before taking traffic;
garbage-collection pauses and noisy neighbours: tune memory; isolate resources;
slow dependencies: set timeouts and fallbacks on every call;
hedged requests: if a call has not returned by the p95 time, send a second copy to another replica and use whichever answers first.

Why it matters

At scale, rare is constant.

A 1-in-1,000 slow request happens about seventeen times a minute at Kirana Kart's peak home-screen traffic.
A page that calls five services is slow if any one of them is, so each service's tail matters more than its average.

In the real world: Google's well-known paper 'The Tail at Scale' introduced hedged requests and showed how much they cut tail latency for a small amount of extra load.

import asyncio

async def hedged(call, replicas, hedge_after_s=0.02):
    """Send to one replica; if slow, also send to another; return the first answer."""
    first = asyncio.create_task(call(replicas[0]))
    done, _ = await asyncio.wait({first}, timeout=hedge_after_s)
    if done:
        return first.result()
    second = asyncio.create_task(call(replicas[1]))
    done, pending = await asyncio.wait({first, second}, return_when=asyncio.FIRST_COMPLETED)
    for task in pending:
        task.cancel()
    return done.pop().result()

Reporting average latency. A service averaging 15 ms with a p99 of 400 ms is a slow service for one user in a hundred.

Interview questions: inference optimisation

What it is

The questions interviewers ask about inference optimisation, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Senior ML interviews often include 'the model is too slow by 3×; what do you do?'. Starting with measurement, not quantisation, is what is listened for.

Q: How would you reduce a model's serving latency?

Profile first. Usually batch feature reads, then batch model calls, then shrink the model (fewer trees, quantisation, distillation, compilation), and add caching and precomputation where inputs repeat.

Q: What is dynamic batching?

The server briefly waits to combine requests from different users into one model call, spreading fixed overhead. It raises throughput, especially on GPUs, at the cost of up to the maximum wait in latency.

Q: What is quantisation's trade-off?

Smaller, faster models using 16- or 8-bit numbers, with a small accuracy risk that must be measured on the evaluation set and slices.

Q: What is distillation?

Training a small student to match a large teacher's predicted probabilities, keeping much of the accuracy at a fraction of the cost.

Q: CPU or GPU for serving?

Compare cost per thousand predictions at real traffic, including the minimum replicas. GPUs win for large neural models at high, steady load; CPUs often win for trees, small models and modest traffic.

Q: How do you autoscale an ML service?

On requests per replica, queue depth or latency rather than only CPU use, with a minimum for reliability and spikes, a maximum for cost, and warm-up so new replicas are ready before they take traffic.

Jumping to GPUs or quantisation without profiling.

Exercise: make a model three times faster

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: Latency reduction projects like this are common first projects for ML engineers on serving teams.

Your task. Take a served model and cut its p99 latency or cost per prediction by at least two thirds, with measured accuracy at each step.

Steps

Start from your FastAPI service in module 18; load-test it and record p50, p99 and requests a second.
Add timing for each step (features, pre-processing, model, logging) and find the largest.
Fix the largest step; re-measure.
Reduce model size (fewer trees, or quantise a neural model) and re-check accuracy on the evaluation set and slices.
Export to ONNX Runtime and re-measure.
For a neural model, try batching 1, 8, 32 and 128 items per call and record throughput and latency.

You are done when

You have a table of latency and accuracy after each change.
p99 latency or cost per prediction is at least three times better than at the start.
No slice lost more accuracy than a limit you set in advance.

Stretch. Add a hedged request between two replicas, inject random slow responses into one, and measure p99 with and without hedging.