What ML System Design Is

ML system design is planning everything around a machine learning model so that it helps a real product, reliably, for years:

• where the data comes from and how it is cleaned;
• how the labels (the right answers it learns from) are collected;
• which features (the facts it looks at) are computed, and when;
• how the model is trained, checked and released;
• how predictions reach the product fast enough and cheaply enough;
• how the system is watched and retrained as the world changes.

A model that scores well in a notebook is not yet useful to anyone:

• it needs fresh inputs every time it is asked something;
• its answers must arrive in milliseconds, or overnight, depending on the product;
• it slowly gets worse as people's behaviour changes, and nothing will crash to tell you.

Design is how you plan for all of that before it goes wrong.

A model is one learned function: inputs go in, a prediction comes out. A system is everything that makes that function useful:

• the pipelines that collect and clean data;
• the code that turns raw data into features, the same way in training and in production;
• the service that answers prediction requests;
• the release process, the dashboards, the alerts and the retraining jobs.

In a typical production system the model's own code is a small fraction of the total.

Most failures in production are system failures, not model failures.

• A feature computed differently in production than in training.
• A data feed that silently stops updating.
• A release with no way to roll back.
• Better modelling fixes none of these.

Ordinary software does what its code says. An ML system does what its data taught it. That changes four things:

• Behaviour comes from data: change the data and you change the product, without touching code.
• It fails quietly: a wrong prediction looks exactly like a right one; nothing throws an error.
• It decays: the world changes, the data it learned from goes out of date, and accuracy slips week by week.
• It is tested with statistics: there is no single correct output to assert, only rates of being right across many cases.

Each difference demands a part ordinary software does not need.

• Data comes from outside your control, so it needs checks of its own.
• Quiet failure needs monitoring of the predictions themselves, not just of the servers.
• Decay needs retraining.
• Statistical testing needs evaluation sets and experiments.

Every ML system, whatever it predicts, has the same parts, joined in a loop:

• Data collection: events and records from the product.
• Labelling: attaching the right answer to past examples.
• Feature pipelines: turning raw data into the model's inputs.
• Training: fitting the model, and checking it against held-back data.
• Model registry: storing each trained version with its scores.
• Serving: answering prediction requests.
• Monitoring: watching inputs, outputs and results.

Monitoring and new data feed back into training, which is what makes it a loop.

Naming the parts gives every design the same skeleton.

• Each later module of this course is one of these parts in depth.
• In an interview, walking the loop once at a high level, before going deep anywhere, shows you see the whole system.

A system is never finished. It goes round a cycle, again and again:

• Scope: what decision does the model help, and how will success be measured?
• Data: collect, label and explore.
• Model: build a baseline, then improve it.
• Deploy: release safely, alongside the old version first.
• Operate: monitor, debug and retrain.

Lessons from operating send you back to any earlier step, including scoping.

Expect to go round many times.

• The first version is usually simple and gets a team most of the value.
• Each later round is driven by what monitoring and error analysis reveal, not by guesses.
• Planning for iteration from the start (versioned data, repeatable training) makes each round cheap.

ML earns its cost only when all of these hold:

• there is a pattern to learn, and it is too complex to write down as rules;
• there is data with that pattern in it, including the right answers;
• mistakes are tolerable: a wrong prediction is a cost, not a disaster;
• the problem repeats at a scale where automation pays;
• the pattern changes over time, so hand-written rules would need constant upkeep.

If a few simple rules get most of the way, ship the rules first.

ML is expensive to build and to keep running.

• It needs data pipelines, training, serving and monitoring, forever.
• A rule is cheaper, explainable and easy to change.
• Rules also make the best baseline: the score an ML model must beat to justify itself.

One example carries this whole course. Kirana Kart is a made-up grocery delivery app with 5,000,000 monthly users, 600,000 orders a day and 400 small warehouses ("dark stores"). It uses machine learning in five places:

• Recommendations: 'you might also like' on the home screen and in the basket.
• Search ranking: ordering results when someone types 'milk'.
• Delivery time (ETA): the minutes shown before someone places an order.
• Fraud detection: spotting fake accounts, stolen cards and refund abuse.
• Demand forecasting: how much of each product each store should stock tomorrow.

The five want different things, which is what makes every design decision a real choice:

• recommendations and search must answer in tens of milliseconds, for millions of requests;
• the ETA must be accurate to the minute and is checked by every customer;
• fraud is rare, its labels arrive weeks late, and fraudsters fight back;
• forecasts are made once a night, but a wrong one wastes food or loses sales.

Almost every decision in ML system design trades between a few forces:

• Accuracy: how often the prediction is right.
• Latency: how long it takes to get one.
• Freshness: how up to date the data behind it is.
• Cost: compute for training and serving, and people's time.
• Simplicity: how easy it is to run, debug and change.

You cannot maximise all of them. A design is a defensible choice of which to favour, for this product.

Naming the trade-off is what interviewers listen for.

• 'A larger model would be more accurate but would take 200 ms, over our 100 ms budget, so I would use it only to rank the top 50' is a design decision.
• 'I would use a deep model' is not.

Every design in this course follows the same eight steps, in order:

• 1. Clarify the problem: the business goal, the users, the scale, the constraints.
• 2. Frame it as ML: what exactly is predicted, from what, and how success is measured.
• 3. Data and labels: what exists, what must be logged, where the right answers come from.
• 4. Features: what the model looks at, and how fresh each input must be.
• 5. Model: a baseline first, then the model that fits the constraints.
• 6. Evaluation: offline measures, then an online experiment.
• 7. Serving: how predictions reach the product, within time and cost.
• 8. Monitoring and iteration: what is watched, and how the model is retrained.

The order matters.

• You cannot choose features before you know what is predicted.
• You cannot choose a model before you know the latency budget.
• You cannot claim success without saying how it is measured.
• The same eight steps work in a forty-five-minute interview and in a two-week design document.

The questions interviewers ask about what ML system design is, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.