ML System Design · Module 23 of 47 · 10 concepts · about 12 min

Debugging Production ML

What to do when a live model goes wrong: limit the damage first, then work through system, data, model and world in order, split a metric change into its causes, reproduce single bad predictions, and learn from each incident with a blameless review.

At work: The skill senior ML engineers are most valued for. Scenario questions in interviews ('your model's accuracy dropped; what do you do?') test exactly this.

Mitigate before you diagnose

What it is

When a live model is causing harm, stop the harm first, then find the cause:

roll back to the previous model if a release is suspected;
switch to the fallback (rules, defaults) if inputs are broken;
raise thresholds or pause an automated action (blocking, sending) while investigating;
communicate: tell affected teams what is happening and what was done.

Why it matters

Diagnosis can take hours; harm accumulates every minute.

A rollback is reversible; lost orders and blocked honest customers are not.
Kill switches and fallbacks (module 20) exist so this step takes seconds.

In the real world: When Kirana Kart's fraud flag rate tripled one evening, the on-call engineer switched to the rules-only policy within five minutes, and the cause (a broken device feature) was found the next morning.

Debugging live while the model keeps blocking customers 'because we are close to finding it'. Mitigate, then investigate calmly with the evidence preserved.

Practise it: Debug a Production ML Model

A triage order

What it is

Work from the most common and cheapest-to-check causes to the rarest:

1. System: did anything deploy? Errors, timeouts, fallbacks?
2. Data: missing, stale or default-filled features? Upstream schema or unit changes?
3. Model: a new model version? Training data problems? Skew?
4. World: real change in behaviour, season, an event, an adversary?

Why it matters

Most incidents are found in the first two steps.

Checking 'what changed recently' (deploys, config, upstream releases) solves a large share of them.
Real-world drift is the most interesting explanation and the least common cause of a sudden drop.

In the real world: Kirana Kart's ML incident runbook begins with three questions: what deployed in the last 24 hours, which features have abnormal null or default rates, and which segments are affected.

Visual example

Triage

Stop at the first step that explains the symptom.

Checking in order of likelihood

What changed?check firstDeploys, models, configs, upstream releases
SystemErrors, latency, timeouts, fallbacks
DataNulls, defaults, freshness, schema
ModelVersion, training data, skew
WorldrarestSeason, event, competitor, adversary

Starting with the most interesting theory. Teams lose days investigating 'changed customer behaviour' when an app release had renamed a field.

When accuracy drops

What it is

A checklist for 'the model got worse':

Sudden or gradual? Sudden points to a bug or release; gradual to drift.
Everywhere or in some segments? Localised points to a specific input, platform or region.
Are the labels right? A change in how labels are recorded can look like a model failure.
Is it the mix? The traffic may have shifted towards harder cases without the model changing at all.
Inputs or relationship? Input drift versus concept drift (module 21).

Why it matters

Each answer narrows the search sharply.

Many 'accuracy drops' turn out to be label changes or traffic-mix changes, with the model unchanged.
Asking these questions out loud is exactly what interviewers want to hear.

In the real world: A team's precision 'dropped' overnight; the cause was the review team changing what they marked as fraud, not the model.

Visual example

Checklist

Start from the shape of the problem.

Symptoms and their likely causes

symptom	likely cause	first check
Sudden drop everywhere	Release or broken input	Recent deploys; feature null and default rates
Sudden drop in one segment	Client or regional data bug	App version, platform, region breakdown
Gradual decline	Drift	Feature PSI; error by week since training
Drop in metric, not in complaints	Label or metric change	How labels are recorded; metric code
Average worse, segments unchanged	Traffic mix shift	Decompose by segment

Assuming the model is at fault. Check the measuring instrument (labels, metric code, traffic mix) before the thing being measured.

Splitting a change into mix and performance

What it is

An overall metric can change for two different reasons:

mix: traffic moved towards segments that are naturally harder (or easier);
performance: the model got worse within segments.

Decompose: mix effect = Σ (new share − old share) × old error; performance effect = Σ new share × (new error − old error). The two add up to the total change.

Why it matters

They need completely different responses.

A mix shift may need nothing at all, or a better model for the growing segment.
A within-segment drop means something broke or drifted.
Without the split, teams 'fix' models that were never broken.

In the real world: Kirana Kart's ETA error rose after a marketing push in small towns; decomposition showed about two-fifths of the rise was mix and the rest a real decline, mostly within small towns, which led to a town-specific feature.

Visual example

Computed

Average ETA error rose from 4 to 4.4 minutes. 0.17 of the rise is mix; 0.23 is the model getting worse within segments.

Share of orders and average error, by segment

segment	share before	share after	error before	error after
Big cities	70%	60%	3.5	3.6
Small towns	30%	40%	5.17	5.6

Decomposition

mix: mix = (0.6 − 0.7) × 3.5 + (0.4 − 0.3) × 5.17 = 0.17

within: performance = 0.6 × (3.6 − 3.5) + 0.4 × (5.6 − 5.17) = 0.23

total: total = 0.17 + 0.23 = 0.4

Comparing overall averages across periods with different traffic mixes. An expansion into small towns makes the average error rise even if the model improved everywhere.

Reproducing a single bad prediction

What it is

To debug one surprising prediction, rebuild it exactly:

find the request by its request ID;
retrieve the logged features and model version (module 5);
re-run that model version on those features: does it give the same output?
then recompute the features from source data: do they match what was logged?
explain the prediction (SHAP) to see which inputs drove it.

Why it matters

Each step isolates a different fault.

Same output from logged features: the model did what it was given; look at the features.
Different output: a serving or version problem.
Logged features differ from recomputed ones: skew or a pipeline bug.

In the real world: A customer complaint about a 90-minute estimate is traced in ten minutes: the logged 'orders waiting' feature was 400, a duplicated-event bug in the stream.

req = prediction_log.get("r-8f2c91")                       # features, model version, output
model = registry.load("eta", version=req.model_version)
assert abs(model.predict([req.features])[0] - req.output) < 1e-6, "serving differs from offline"

recomputed = training_features.for_request(req.order_id, as_of=req.ts)
diffs = {k: (req.features[k], recomputed[k]) for k in req.features if req.features[k] != recomputed[k]}
print(diffs)   # e.g. {'hour_of_day': (19, 13)} -> a time-zone bug in the training pipeline

Debugging from the features recomputed today. They may differ from what the model saw at the time, which hides the very bug you are looking for.

Golden sets and canary inputs

What it is

A golden set is a fixed set of inputs with known, checked outputs, scored automatically:

on every new model version, before release;
on the live service, every few minutes, as synthetic requests;
alerting if the outputs change unexpectedly.

Why it matters

It catches silent changes quickly.

A library upgrade, a serving change or a corrupted model alters golden outputs immediately.
Synthetic requests also check the full live path, features included, when real traffic is quiet.

In the real world: Kirana Kart sends twenty fixed ETA requests through the live service every five minutes and alerts if any answer moves by more than two minutes without a model release.

A golden set that never changes while the product does. Keep it representative: add cases from incidents and new segments.

Common root causes

What it is

Across many teams, ML incidents trace back to a short list:

upstream data changes: renamed fields, new units, new categories;
pipeline failures: missing or late partitions, stalled streams;
training-serving skew: different code or defaults;
bad releases: a model trained on broken data, or the wrong model loaded;
drift: the world changed;
feedback loops and adversaries.

Why it matters

Knowing the list speeds up every investigation.

The first three are the most common and the cheapest to prevent with contracts, validation and logged features.
Each incident should end with a check that would have caught it earlier.

In the real world: Reviews of ML incidents at large companies consistently find that data problems, not model problems, are the leading cause.

Fixing the symptom and moving on. Without a new check, the same class of failure returns in a few months under a different name.

Blameless postmortems

What it is

After an incident, write a short blameless postmortem:

what happened, with a timeline;
impact: users, orders, money, duration;
root cause and contributing factors;
what went well and badly in detection and response;
actions: checks, alerts or process changes, each with an owner and date.

Blameless means focusing on how the system allowed the mistake, not on who made it.

Why it matters

Postmortems turn incidents into lasting improvements.

Blame makes people hide mistakes; blamelessness surfaces them.
The action list is the point: each one closes a gap.

In the real world: After the stalled-stream incident, Kirana Kart added a freshness alert on every streaming feature and a rule that the ETA service falls back when the queue feature is more than five minutes old.

A postmortem whose actions are 'be more careful'. Every action should be a concrete check, alert, test or automation.

Interview questions: debugging production ML

What it is

The questions interviewers ask about debugging production ML, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Scenario questions like these appear in most senior ML interviews; a calm, ordered method matters more than guessing the exact cause.

Q: Your model's accuracy dropped. What do you do?

Mitigate first if users are harmed: roll back or fall back. Then ask what changed recently, check system health, then data (nulls, defaults, freshness, schema), then the model (version, skew), and only then real-world drift. Check whether it is sudden or gradual, everywhere or in segments, and whether labels or traffic mix changed.

Q: How do you tell traffic mix from model decline?

Decompose the change by segment into a mix effect, from shares moving, and a performance effect, from error changing within segments.

Q: How do you debug a single bad prediction?

Fetch the logged features and model version by request ID, re-run them to check serving, recompute the features from source to check for skew, and explain the prediction with SHAP.

Q: The model was great offline but poor from day one. Why?

Most likely training-serving skew or leakage in training; run a replay test comparing logged and recomputed features, and check for features unavailable at prediction time.

Q: What is a golden set?

Fixed inputs with checked outputs, scored on every release and periodically against the live service, to catch silent changes.

Q: What goes in a postmortem?

Timeline, impact, root cause, what helped or hindered response, and concrete actions with owners, written without blame.

Jumping straight to 'retrain the model'.

Practise it: Debug a Production ML Model

Exercise: run an incident drill

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: Some teams run drills like this for new on-call engineers before they join the rotation.

Your task. Diagnose three planted faults in a running model service using only logs and monitoring.

Steps

Ask a friend (or write a script) to plant one fault at a time in your service from earlier modules, without telling you which: (a) a feature filled with zeros, (b) a model trained with a time-zone bug, (c) traffic shifted towards a harder segment with no model change.
For each, write down the symptom you see first.
Follow the triage order and record each check and its result.
Use request-level reproduction for at least one bad prediction.
Use mix decomposition where the overall metric changed.
Write a one-page blameless postmortem for one of them with at least two concrete actions.

You are done when

You correctly identify all three faults.
For the mix shift, you show the model did not get worse within segments.
Each postmortem action is a check or alert, not 'be careful'.

Stretch. Time yourself and repeat with new faults until each diagnosis takes under fifteen minutes.