Monitoring and Drift

Watch a live model at four layers, from the machine up to the business:

• System: requests, errors, latency, resource use. Is the service healthy?
• Data: input features: missing values, ranges, freshness, distributions. Are the inputs normal?
• Model: predictions and, when labels arrive, accuracy. Is the model behaving as expected?
• Business: the outcomes it exists for: orders, fraud losses, complaints. Is it still helping?

Each layer catches failures the others miss.

• A perfectly healthy service can serve terrible predictions.
• Inputs can drift while accuracy holds, or accuracy can fall while inputs look normal (concept drift).
• Business metrics are the final judge, but slow and noisy, and moved by many things besides the model.

Drift is change over time that makes a model less accurate. Four kinds:

• Data drift (covariate shift): the inputs change. Bigger baskets during a festival; a new city with longer distances.
• Label drift (prior shift): how common each outcome is changes. Fraud rises from 0.5% to 1.5% during an attack.
• Concept drift: the relationship between inputs and outcome changes. The same distance now takes longer because of new road works; fraudsters change tactics.
• Prediction drift: the model's outputs change, a symptom of any of the above, or of a bug.

The kind of drift decides the response.

• Data drift into ranges the model has seen may be harmless; into new ranges it is risky.
• Label drift can often be handled by recalibrating or moving the threshold.
• Concept drift needs retraining on recent data, and sometimes new features.
• Sudden changes usually mean a bug or an event; gradual ones mean the world is moving.

Common ways to compare a feature's live distribution with its training distribution:

• Population stability index (PSI): bin the values, compare each bin's share. Under 0.1 is usually stable, 0.1–0.25 a moderate shift, over 0.25 large.
• Kolmogorov–Smirnov test: the largest gap between two cumulative distributions, for numbers.
• Chi-square test: for categories.
• Jensen–Shannon or Wasserstein distance: smooth measures of how far apart two distributions are.

A number per feature makes drift visible and comparable.

• PSI is popular because it is simple, works for any binned feature, and shows which bins moved.
• Statistical tests become 'significant' for tiny, harmless shifts once samples are large, so a size measure like PSI is often more useful than a p-value.

Labels often arrive late or never. Watch what you can see now:

• prediction drift: the distribution of scores or outputs;
• input drift, weighted by how important each feature is to the model;
• proxy outcomes that arrive fast: complaints, manual overrides, review decisions, early returns;
• confidence: a rise in uncertain predictions suggests unfamiliar inputs.

Waiting for labels can mean months of unnoticed decline.

• These signals do not prove accuracy has fallen, but they say where to look.
• Weighting drift by feature importance avoids alarms about features the model barely uses.

When labels arrive over weeks, measure performance by cohort:

• group predictions by the week they were made;
• for each cohort, compute accuracy on the labels arrived so far;
• compare each cohort with older cohorts at the same age, not with fully labelled ones.

Recent cohorts always look different, because their labels are incomplete.

• Comparing a 7-day-old cohort with a 90-day-old one shows a fake decline.
• Comparing it with how earlier cohorts looked at 7 days shows a real one.

The checks that guard training data (module 8) should also run on live features:

• share of missing values per feature;
• share of default values (0, -1, 'unknown'): a jump means an upstream source broke;
• ranges and allowed categories;
• freshness: age of each online feature;
• volume: requests per segment, compared with the usual pattern.

Live data breaks more often than models decay.

• A sudden jump in defaults for one feature is the most common sign of a broken upstream service.
• These checks are cheap, fast and precise about the cause.

Monitor key metrics per segment, not only overall:

• by city, store, device, app version, customer type;
• especially segments that matter commercially or are vulnerable (new users, small cities);
• with enough volume per segment for the numbers to mean something.

Many failures are local.

• A new app version on one platform sends a feature in the wrong units: overall accuracy dips slightly; that platform's accuracy collapses.
• Averages dilute such failures until they are invisible.

An alert is useful only if someone acts on it. Good alerting:

• alerts on symptoms that matter (errors, stale features, prediction drift, business impact), not every metric;
• requires a condition to hold for a while (several consecutive checks) before firing, to ignore blips;
• groups related alerts (one upstream failure, one alert, not fifty);
• ranks by impact: traffic affected, money at risk;
• links to a runbook: what to check and what to do.

Too many alerts are as bad as none.

• People learn to ignore a noisy channel, and then miss the real incident.
• Every alert should be either actionable or removed.

Storing every prediction and feature to compute distributions is expensive at millions of requests an hour. Sketches summarise a stream in a small, fixed amount of memory:

• quantile sketches (t-digest, KLL) estimate percentiles;
• HyperLogLog estimates the number of distinct values;
• frequency sketches (count-min) estimate how often values appear.

Sketches from many servers and time windows can be merged.

They make per-feature, per-segment monitoring affordable.

• A few kilobytes per feature per window instead of every value.
• Accuracy is good enough for drift detection.

Every live model needs clear ownership:

• a named owning team and on-call rotation;
• dashboards for the four layers;
• runbooks for common incidents: stale feature, drift, bad release;
• a review cadence: monthly look at performance, slices, alerts and costs.

Models without owners decay quietly.

• The person who built it moves on; nobody notices its slow decline.
• Clear ownership is the cheapest monitoring there is.

The questions interviewers ask about monitoring and drift, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.