ML System Design · Module 47 of 47 · 11 concepts · about 15 min

Interview: Scenario and Debugging Questions

Ten situations interviewers describe, from a sudden accuracy drop to a model that wins offline and loses online, each answered with one method: clarify, limit the damage, check in order, find the cause, fix it and stop it recurring.

At work: Standard in senior ML interviews and common in on-call and MLOps roles. There is rarely one right cause; interviewers score the method.

A method for scenario questions

What it is

Answer every 'something is wrong' question with the same six steps:

clarify: what exactly changed, since when, how measured, for whom;
mitigate: if users are harmed, roll back or fall back first (module 23);
triage in order: recent changes, system, data, model, world;
diagnose: narrow with segments, decomposition, replay tests and single-prediction reproduction;
fix: the cause, not the symptom;
prevent: the check or alert that would have caught it sooner.

Why it matters

A method keeps you calm and complete under pressure.

It shows the interviewer you would be safe to put on call.
Naming several plausible causes, and how you would tell them apart, scores better than guessing one.

In the real world: Interviewers report that the best answers sound like an incident runbook: questions first, harm limited, causes checked in order of likelihood.

Visual example

Method

The same order every time.

Six steps for any scenario

ClarifyWhat, since when, for whom
MitigateRoll back or fall back
TriageChanges, system, data, model, world
DiagnoseSegments, replay, one case
FixThe root cause
PreventAn alert for next time

Leaping to a single cause. Interviewers often add 'and that's not it' to see whether you have a method or only a guess.

Accuracy dropped suddenly overnight

What it is

'Our recommendation model's click-through fell 15% overnight. What do you do?'

Why it matters

A strong answer, step by step:

clarify: all platforms or one? all users or some? did anything deploy (app, model, pipeline)?
mitigate: if a model release coincides, roll back while investigating;
check data: null and default rates per feature, freshness of online features, schema changes from an app release;
check serving: fallback rate, errors, latency, whether the right model version is loaded;
check measurement: did click logging change in the app?
prevent: data-quality alerts per feature; golden-set checks after each release.

In the real world: A real case: an app release renamed a field; the 'items in basket' feature became zero for Android users, and click-through fell only on Android.

A model answer, said aloud

“A sudden overnight drop usually means something changed, not that users did. First I'd ask what deployed in the last day, model, pipeline or app, and whether the drop is everywhere or on one platform. If a model release lines up, I'd roll back immediately. Then I'd look at feature null and default rates and freshness, since a broken upstream source is the most common cause, and at the fallback rate in serving. I'd also check that click logging itself didn't change in a new app version. Once found, I'd add the alert that would have caught it within minutes.”

Proposing to retrain. Retraining on a day of broken data makes things worse, and a sudden drop is rarely drift.

Accuracy is declining slowly

What it is

'Our delivery-time error has crept up 2% a month for six months. What's going on?'

Why it matters

A strong answer, step by step:

clarify: everywhere or in some cities, stores, hours? how often is it retrained, and on what window?
decompose: traffic-mix change (new cities, more evening orders) versus worse performance within segments (module 23);
check drift: feature PSI over time; concept drift such as new traffic patterns or store layouts;
check retraining: is it actually running, with a sensible window, and passing the gate?
fix: retrain more often or on a shorter window, add features for the new situation;
prevent: decay curve monitoring and a performance trigger.

In the real world: Teams often discover their 'nightly retrain' had silently failed weeks earlier, leaving an old model in production.

A model answer, said aloud

“Slow decline points to drift or a growing mix of harder cases, not a bug. I'd first split the change into mix and within-segment parts: if we expanded to smaller towns, the average can rise with the model unchanged. Then I'd check whether retraining is actually happening and on what window, and look at feature drift and at which segments got worse. If it's concept drift, say new traffic patterns, I'd retrain on more recent data and add a feature that captures the change. Then I'd monitor error against weeks since training and add a performance trigger.”

Assuming the model decayed without checking the traffic mix. Expansion into harder segments is a very common cause.

Great offline, poor online

What it is

'The new ranker improved NDCG by 4% offline, but the A/B test shows no gain. Why?'

Why it matters

A strong answer, step by step:

check the experiment: sample ratio, exposure, power, duration, novelty;
check skew: replay test of served versus offline features; is the same model version serving?
check latency: did the new model slow the page and cost engagement?
check the metric: does offline NDCG on logged clicks reflect the online goal? data shaped by the old ranker can favour it;
check slices: gains in one group cancelled by losses in another;
prevent: track offline-online agreement across launches; add replay tests before A/B tests.

In the real world: A well-known pattern: offline gains on 'popular items users would have bought anyway' that do not change behaviour online.

A model answer, said aloud

“I'd first make sure the test itself is sound: sample ratio, enough power, long enough. Then I'd look for training-serving skew with a replay test, and check p99 latency, because a slower page can erase a relevance gain. Then I'd question the offline metric: NDCG on clicks logged under the old ranker favours items it already showed, and may not track basket additions. Finally I'd check slices for gains and losses cancelling out. Whatever the cause, I'd start tracking how well offline wins predict online wins.”

Tuning harder on the offline metric. If it doesn't predict online results, a bigger offline gain won't either.

The fraud model is blocking honest customers

What it is

'Customer support says honest customers are being blocked far more than usual. What do you do?'

Why it matters

A strong answer, step by step:

clarify: which customers, since when, which decision (block or review)?
mitigate: raise the block threshold or send blocks to review temporarily; consider the kill switch to rules-only;
check inputs: a feature broken for some customers (device, location) can push scores up;
check the model: new version? recalibration needed after a shift in base rate?
check fairness: is one segment (a new city, an app version) disproportionately affected?
prevent: monitor flag rate and appeal overturns by segment, with alerts.

In the real world: A device-fingerprint library update made many new phones look like emulators; flag rates for those phones jumped overnight.

A model answer, said aloud

“False declines cost us customers, so I'd limit harm first: send would-be blocks to review or raise the block threshold while investigating. Then I'd find who is affected. If it's one segment, a new app version or city, I'd look for a broken feature there, such as device data arriving empty. If it's everyone, I'd check for a recent model release or a shift that broke calibration. Afterwards I'd add per-segment monitoring of flag rate and appeal overturns.”

Defending the model because recall looks good. Recall says nothing about the honest customers who were blocked.

Latency doubled after a release

What it is

'After shipping the new model, p99 latency went from 60 ms to 120 ms. What now?'

Why it matters

A strong answer, step by step:

mitigate: roll back if the budget is breached and users are affected;
profile: feature fetching, pre-processing, model, logging, each at p99 (module 19);
common causes: new features needing extra lookups, a larger model, lost batching, cold starts, a different runtime;
fix: batch lookups, shrink or compile the model, warm up replicas, adjust autoscaling;
prevent: a latency check in the release gate and canary analysis on p99.

In the real world: A common cause in practice is a new feature fetched in a per-candidate loop instead of one batched call.

A model answer, said aloud

“I'd roll back if we're over budget, then profile the new version step by step at p99. Often it's not the model but new features: two extra online-store lookups made one at a time can add 30 ms. If it is the model, I'd look at fewer trees, quantisation or compiling it, and make sure candidates are scored in one batch. And I'd add p99 latency to the release gate so this is caught in the canary next time.”

Adding replicas. More copies raise throughput but rarely fix per-request latency.

A feature's distribution shifted

What it is

'Drift monitoring says the basket-value feature's PSI is 0.3 this week. Should we retrain?'

Why it matters

A strong answer, step by step:

clarify: sudden or gradual? one segment or all? is there a known event (festival, sale, price change)?
check for a bug first: units, defaults, a pipeline change;
check impact: has accuracy or a fast proxy moved? how important is this feature to the model?
decide: a known, temporary event may need nothing; a lasting change needs retraining with recent data;
prevent: weight drift alerts by feature importance; add event calendars to monitoring.

In the real world: Festival weeks routinely trigger drift alerts on basket features at grocery companies; teams pre-annotate them so on-call engineers are not paged needlessly.

A model answer, said aloud

“Not automatically. I'd first rule out a bug, such as a unit change or defaults, because retraining on a bug teaches it to the model. Then I'd ask whether there's an obvious cause, like a festival sale, and whether accuracy or proxies have actually moved. If the shift is temporary and performance is fine, I'd leave it. If it's lasting and hurting performance, I'd retrain on recent data. I'd also make sure drift alerts are weighted by how much the model relies on each feature.”

Retraining on every drift alert. Some drift is harmless, and some is a bug.

Predictions are good overall but bad for a group

What it is

'Our ETA model is accurate overall, but customers in one new city complain constantly. What do you do?'

Why it matters

A strong answer, step by step:

measure: error and lateness for that city against others; how much training data does it have?
look for causes: different road patterns, store layouts, missing features (no rider data yet), label problems;
fix: more weight or data for the city, city-specific features, a transfer approach from similar cities, or a more conservative percentile there;
evaluate per slice in the gate so new cities cannot regress unseen;
prevent: slice-level monitoring and launch checklists for new regions.

In the real world: New-region underperformance is one of the most common complaints after an expansion; slice monitoring is the standard fix.

A model answer, said aloud

“Overall averages hide small segments. I'd measure the city's error and lateness separately, check how much data we have there and whether its labels are reliable. New cities often lack history, so features like the store's average packing time are thin. I'd borrow from similar cities, add city-level features, and in the meantime promise a higher percentile there. Then I'd add per-city checks to the release gate and monitoring.”

Telling the interviewer the model is fine because the overall error is low.

Recommendations all look the same

What it is

'Users say recommendations are repetitive: the same popular items for everyone. Why, and how would you fix it?'

Why it matters

A strong answer, step by step:

diagnose: measure catalogue coverage and diversity over time; check for a feedback loop (module 22);
check features: is personalisation actually reaching the ranker (stale or missing user features)?
check candidates: is one source (popularity) dominating?
fix: exploration, source quotas, diversity in re-ranking, debiasing for popularity and position;
measure: diversity and coverage metrics alongside engagement in experiments.

In the real world: Teams have found 'repetitive recommendations' caused by an expired online-store key format that left every user with default features.

A model answer, said aloud

“Two common causes: a feedback loop, where the model learns from its own exposure and narrows onto popular items, or personalisation silently failing, for example user features missing, so the ranker falls back to popularity. I'd check user-feature freshness first, then coverage over time and the share of candidates from each source. Fixes are exploration with logged probabilities, quotas per candidate source, diversity in re-ranking and debiasing in training, measured with coverage metrics next to engagement.”

Only adding randomness. Diversity without relevance lowers engagement; fix the cause and measure both.

No labels for a new model

What it is

'We want to detect a new kind of abuse, but have no labelled examples. How do you start?'

Why it matters

A strong answer, step by step:

rules and heuristics from experts as a first defence and a source of weak labels;
unsupervised anomaly detection to surface unusual cases;
human review of a targeted sample, with written guidelines and agreement measured;
active learning to pick the next cases to label;
a supervised model once a few thousand labels exist, evaluated on a clean random sample.

In the real world: Trust-and-safety teams routinely launch with rules and anomaly detection, then move to supervised models as reviewer labels accumulate.

A model answer, said aloud

“I'd start with rules from people who understand the abuse, which also act as weak labels, and an anomaly detector to surface unusual patterns. A small review team labels a targeted sample under clear guidelines, and active learning picks the most informative next cases. Once we have a few thousand good labels, I'd train a supervised model and keep a random, carefully labelled sample to measure precision and recall honestly.”

Waiting months to collect labels before doing anything.

The model must be ten times cheaper

What it is

'Serving cost for our ranking model is too high. Cut it by 10×.'

Why it matters

A strong answer, step by step:

measure where the cost goes: which model, how many calls, hardware use, idle capacity;
reduce calls: cache, precompute in batch, add a light pre-ranker so the heavy model sees fewer items;
reduce cost per call: distil, quantise, prune, fewer features, compile;
right-size: CPU versus GPU at real traffic, autoscaling, scale-down at night;
check value: is the model worth it compared with a simpler one? measure with an experiment.

In the real world: Teams frequently find that a pre-ranking stage plus batch precomputation cuts heavy-model calls by most of the total.

A model answer, said aloud

“First I'd find where the money goes. The biggest wins usually come from not calling the heavy model as much: caching popular requests, precomputing for active users overnight, and adding a light pre-ranker so the heavy model scores 100 items instead of 500. Then making each call cheaper with distillation and quantisation, and right-sizing hardware with autoscaling. Combined, 10× is realistic. I'd also check, with an experiment, whether a much simpler model loses anything users notice.”

Only quantising. A single technique rarely gives 10×; combining fewer calls with cheaper calls does.