ML System Design · Module 2 of 47 · 11 concepts · about 15 min

The Machine Learning You Need

A plain refresher on the ideas every later module relies on: what a model learns, training versus prediction, the kinds of task, features and labels, overfitting, the main model families, and turning a score into a decision.

At work: The shared vocabulary of every ML design meeting. Engineers joining an ML team from ordinary software need exactly this much to follow the conversation.

What a model learns

What it is

A model is a function learned from examples rather than written by hand:

you show it many past cases, each with its inputs and the right answer;
training adjusts the model's internal numbers (its parameters) until its answers on those cases are as close to right as possible;
afterwards it can give an answer for a new case it has never seen.

Why it matters

This is why data quality matters more than anything else.

The model can only learn patterns that are in its examples.
If the examples are biased, out of date or wrongly labelled, the model learns that faithfully.
'Garbage in, garbage out' is the oldest rule in the field for good reason.

In the real world: Kirana Kart's delivery-time model learns from a year of past orders, each with its distance, store, time of day and the minutes it really took.

Imagining the model 'understands' the task. It finds whatever pattern best predicts the labels it was given, including accidental ones, such as a hospital's scanner model that learned to read the hospital's name printed on the image.

Training and inference

What it is

A model has two phases, and they run in very different places:

Training: learning from many past examples at once. Slow, heavy, usually done offline on a schedule.
Inference (or prediction, or serving): answering for one new case, or a batch of them. Fast, light, done constantly.

Why it matters

Most system design is about the gap between the two.

Training sees a tidy table of history; inference sees one live request.
The inputs must be computed the same way in both, or the model meets data it was never trained on.
Training cost is paid once per version; inference cost is paid on every request, forever, so it usually dominates.

In the real world: Kirana Kart retrains its recommendation model once a night on yesterday's clicks, then serves it eight million times the next day.

Visual example

Two phases

The same model, used in two different ways.

Training and inference compared

	training	inference
Input	Millions of past examples	One new case (or a batch)
Output	A trained model	A prediction
How often	Daily, weekly, or on a trigger	Every request
Speed that matters	Hours are fine	Milliseconds
Where	Training cluster, often GPUs	Prediction service, CPUs or GPUs
Cost pattern	Once per version	Every request, forever

Optimising training time while ignoring inference. A model that trains in an hour but takes 300 ms per prediction may be unusable in a product with a 100 ms limit.

Practise it: Design a Predictive Maintenance System, Design an ML Training Pipeline, Design Feature Monitoring, Diagnose Training-Serving Skew, Design an ML Feature Store

Kinds of ML task

What it is

Most ML products are one of a handful of task types:

Classification: which category? Fraud or not; spam or not; which department.
Regression: what number? Delivery minutes; price; time to failure.
Ranking: in what order? Search results; recommendations; a feed.
Forecasting: what number, in the future, over time? Tomorrow's demand per store.
Clustering and anomaly detection: which items are alike, and which are unusual? Without labels.
Generation: produce text, images or code. The subject of the LLM topics.

Why it matters

The task type decides most of what follows.

It sets the label you need, the loss the model learns from and the metrics you report.
Many products can be framed in more than one way, and choosing the framing is one of the most important design decisions (module 3).

In the real world: Kirana Kart has one of each: fraud (classification), delivery time (regression), recommendations and search (ranking), stock levels (forecasting).

Visual example

Task types

Later modules explain each measure in full.

Each task type, with its label and a typical measure

task	predicts	label example	typical measure
Classification	A category or a probability	Was this order fraud? yes/no	Precision, recall, PR-AUC
Regression	A number	Minutes the delivery took	Mean absolute error
Ranking	An order of items	Which item was clicked	NDCG, recall at k
Forecasting	Future numbers over time	Units sold each day	WAPE, pinball loss
Anomaly detection	How unusual a case is	Often none	Precision of alerts reviewed

Framing a ranking problem as classification and stopping there. 'Will the user click this item?' is a classifier, but the product shows a list, and what matters is whether the best items are at the top.

Features and labels

What it is

Each training example has two halves:

Features: the facts the model is given. Distance to the customer, items in the order, hour of day.
Label: the right answer it should predict. The minutes the delivery actually took.

At prediction time only the features exist; the label is what the model supplies.

Why it matters

Two rules follow, and most production bugs break one of them.

Every feature must be available at prediction time, computed from information that exists at that moment.
No feature may contain the label in disguise. 'Refund issued' cannot be a feature for predicting fraud if refunds are issued after fraud is confirmed.

In the real world: A churn model scores 99% in testing because one feature was 'days since last login', computed today for customers who had already left. In production it predicts nothing.

Leakage: a feature that is only known after the event. It makes the model look superb in testing and useless in production, because the feature is not there yet when the real prediction is made.

Practise it: Design a Demand Forecasting Platform, Design a Predictive Maintenance System, Design Model Monitoring, Design a Real-Time Fraud Detection System, Design a Search Ranking System, Design an ML Feature Store

Loss: what the model is trying to minimise

What it is

Training needs a single number that says how wrong the model is: the loss. Training changes the parameters to make it smaller. Common choices:

Log loss (cross-entropy) for probabilities: punishes confident wrong answers hard.
Squared error for numbers: punishes big misses much more than small ones.
Absolute error for numbers: treats every minute of error the same.
Ranking losses for order: rewards putting the right items above the wrong ones.

Why it matters

The loss is what the model actually optimises, so it should match what the business cares about.

A delivery estimate that is 20 minutes late costs more goodwill than one that is 20 minutes early. A loss that treats both the same will not learn that.
The business metric (orders, revenue, complaints) is what you finally care about; the loss is a stand-in the model can learn from.

In the real world: A food-delivery firm trains its delivery model to predict a high-ish estimate (the 80th percentile) rather than the average, because customers forgive early arrival and complain about late arrival.

Assuming a lower loss means a better product. A model can reduce average error by getting easy cases slightly better while getting the important rare cases worse.

Practise it: Design a Demand Forecasting Platform

Overfitting and generalisation

What it is

A model overfits when it memorises its training examples instead of learning the general pattern. To catch it, data is split three ways:

Training set: what the model learns from.
Validation set: what you check against while choosing settings.
Test set: touched once, at the end, for an honest final score.

A model that does far better on training data than on validation data is overfitting.

Why it matters

The only score that matters is on data the model has not seen, because every real prediction is on unseen data.

More capacity (a bigger model) fits training data better and can generalise worse.
More and better data is usually the strongest cure.

In the real world: A team's demand model looks excellent with a random split and fails in production; re-tested with 'train on January to June, test on July', its true error is three times higher.

Splitting time-ordered data at random. If tomorrow's orders are in the training set, the model is tested on a past it has partly seen. For anything that changes over time, train on the past and test on a later period (module 7).

Model families at a glance

What it is

Four families cover most production systems:

Linear and logistic regression: a weighted sum of features. Fast, simple, easy to explain; a strong baseline.
Gradient-boosted trees (XGBoost, LightGBM, CatBoost): many small decision trees added together. The best default for tables of numbers and categories.
Neural networks: layers of learned functions. Best for images, text, audio, and very large datasets with rich interactions.
Embedding models: neural networks that turn items, users or text into lists of numbers, so similar things land close together. The basis of modern retrieval (module 13).

Why it matters

The family is chosen by the data and the constraints, not by fashion.

For tabular data with up to a few million rows, boosted trees usually win on accuracy and are cheap to serve.
Neural networks earn their cost on unstructured data or at very large scale.
Module 12 covers the choice in depth.

In the real world: Most fraud and credit-risk models at banks are gradient-boosted trees; most image and text features at the same banks use neural networks.

Choosing deep learning for a table of thirty columns. It is usually slower to train, harder to tune and no more accurate than a boosted-tree model.

From a score to a decision

What it is

A classifier usually outputs a score between 0 and 1. The product needs a decision, so you choose a threshold:

score at or above the threshold: act (block the order, send for review);
below it: do not act.

Moving the threshold trades one kind of mistake for the other.

Why it matters

The threshold is a business decision, not a modelling one.

A high threshold flags few cases, mostly correctly, but misses many real ones.
A low threshold catches more real cases but flags more innocent ones.
The right point depends on what each mistake costs (module 16 computes it).

In the real world: Kirana Kart uses two thresholds on its fraud score: block automatically above the top one, and send to a human reviewer between the two.

Visual example

Computed

Eight orders scored by a fraud model; four were really fraud. Lowering the threshold catches more fraud and flags more innocent orders.

Scores: 0.95✓ 0.85✓ 0.70 0.60✓ 0.40 0.30 0.20✓ 0.05 (✓ = really fraud)

threshold	orders flagged	fraud caught	share of fraud caught	share of flags that were fraud
0.8	2	2	2 of 4	100%
0.5	4	3	3 of 4	75%
0.15	7	4	4 of 4	57%

The last two columns are recall and precision, defined properly in module 16.

# A fraud score becomes a decision only once a threshold is chosen.
score = model.predict_proba(order_features)[0, 1]   # e.g. 0.73

if score >= 0.9:
    action = "block"
elif score >= 0.6:
    action = "send_to_review"
else:
    action = "allow"

Leaving the threshold at 0.5 because it is the default. Nothing about 0.5 is special; for rare events like fraud, the best threshold is often far from it.

Ways a model can learn

What it is

Models learn from different kinds of signal:

Supervised: from examples with right answers. Most production ML.
Unsupervised: from data with no answers, finding structure: clusters, unusual points.
Self-supervised: inventing answers from the data itself, such as hiding a word and predicting it. How embeddings and language models are trained.
Reinforcement and bandits: from rewards after acting, learning which choice pays off. Used for exploration in recommendations and ads.

Why it matters

The available signal decides the approach.

No labels, and you need anomalies: unsupervised.
Plenty of unlabelled text or images: self-supervised embeddings, then a small supervised model on top.
A choice whose result is observed immediately (a click): bandits can learn while serving.

In the real world: Kirana Kart's fraud system combines a supervised model trained on confirmed fraud with an unsupervised detector that flags new patterns no label has described yet.

Visual example

Signal

Pick by the signal you have.

Choosing how a model learns, by the signal you have

which signal do you have?
Right answersSupervised learning
Lots of raw dataSelf-supervised, then a small model
No answers; want the unusualAnomaly detection
Reward right after actingBandits or reinforcement learning

Assuming labels exist. Many designs stall because nobody asked where the right answers would come from; module 6 is about exactly that.

Interview questions: the ML basics a system designer needs

What it is

The questions interviewers ask about the ML basics a system designer needs, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: These are the warm-up questions in most ML design rounds. Interviewers use them to decide how deep to go later.

Q: What is the difference between training and inference?

Training learns parameters from many past examples, offline and slowly; inference applies the trained model to new cases, constantly and quickly. Inference cost recurs on every request, so it usually dominates.

Q: What is data leakage?

A feature that would not be known at prediction time, or that encodes the label. It makes offline scores look excellent and production performance collapse.

Q: How do you detect overfitting?

Compare training and validation performance; a large gap means the model memorised its training data. The cure is more data, a simpler model, or regularisation.

Q: Why not always use deep learning?

On tabular data, boosted trees are usually as accurate, cheaper to train and serve, and easier to debug. Deep learning earns its cost on images, text, audio, or very large data.

Q: How do you choose a classification threshold?

From the cost of each kind of mistake: weigh false positives against false negatives and pick the threshold that minimises total cost, or meets a required precision or recall.

Q: What is the difference between a loss and a business metric?

The loss is what training minimises and must be differentiable or at least computable per example; the business metric is what the company cares about, like revenue or complaints. The loss is a stand-in for it.

Exercise: train, split and threshold a small model

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: Teams onboarding software engineers into ML often start with exactly this notebook; it makes every later design conversation concrete.

Your task. Train a classifier on a public dataset and see training versus inference, overfitting and thresholds for yourself.

Steps

Load a tabular dataset with a yes/no label (for example scikit-learn's breast cancer data, or a credit default dataset).
Split it into training, validation and test sets (60/20/20).
Train a logistic regression and a gradient-boosted tree model; record each one's score on training and validation data.
Train a deliberately over-large tree (no depth limit) and compare its training and validation scores.
For the best model, print precision and recall on validation data at thresholds 0.3, 0.5 and 0.7.
Time a single prediction, and a batch of 1,000.

You are done when

You can point to one model whose training score is much higher than its validation score, and say why.
You have a table of precision and recall at three thresholds, and can say which you would pick for a fraud use and why.
You know how many milliseconds one prediction takes, and how much faster per item a batch is.

Stretch. Add one leaky feature on purpose (a copy of the label with a little noise) and watch the validation score become unbelievable.

Practise it: Diagnose Training-Serving Skew