Machine Learning · Module 26 of 26 · 12 concepts · about 24 min

Monitoring & Model Drift

Why a model that changed nothing gets worse anyway, how to see it happening before the labels arrive, and what to do when you do.

At work: Noticing when a model quietly gets worse because the world changed, and fixing it before customers do.

Why a deployed model decays

What it is

The model is a frozen description of a relationship in the world at the time it was fitted. The world keeps moving: prices change, a competitor launches, a marketing campaign brings a different population, a product change alters behaviour, an upstream schema changes, fraudsters adapt.

Why it matters

It reframes maintenance as expected rather than exceptional. A model is not a build artifact that stays correct once it passes tests; it is closer to a perishable estimate with a shelf life, and the shelf life is a property of the domain rather than of the code.

In the real world: Customer behaviour shifts after a competitor's launch, and last year's churn model slowly gets worse.

Visual example

Performance decay

Two identical launches: one domain decays in weeks, the other in years. The code did not change in either.

Fraud model (adversarial)Credit risk model (stable)Fraud, retrained every 4 weeks

Decay rates, per week

fraud: -0.0085 AUC/week — crosses the floor at week 11

credit: -0.0006 AUC/week — 80 weeks to lose what fraud loses in 6

retrained: sawtooth between 0.80 and 0.82: never approaches the floor

Same architecture, same rigour. The difference is entirely the domain

Why this matters

The retraining cadence is not a matter of taste; it is set by the decay slope and the distance to the floor. Measuring the slope once tells you what the cadence has to be.

Common misconception

Wrong idea: a model degrades because something broke, so a stable system means a stable model.

Correct idea: it degrades because the world moved while the model stood still. Nothing broke on the fraud curve above — the fraudsters simply adapted, which is what they do.

import pandas as pd

# Performance by cohort month: the shape of decay, measured rather than assumed.
scored = pd.read_parquet('scored_history.parquet')     # prediction, label, scored_at
scored['month'] = scored['scored_at'].dt.to_period('M')

monthly = scored.groupby('month').apply(
    lambda g: pd.Series({
        'n': len(g),
        'auc': roc_auc_score(g['label'], g['p']),
        'mean_p': g['p'].mean(),
        'base_rate': g['label'].mean(),
    }),
    include_groups=False,
)
monthly['auc_vs_launch'] = monthly['auc'] - monthly['auc'].iloc[0]
monthly.tail(6)

Decay rates differ by orders of magnitude between domains, and assuming yours is slow is the mistake. Fraud and ad ranking can degrade meaningfully in days; a medical imaging model may hold for years. Measure your own curve from the first month of production rather than adopting somebody else's retraining cadence.

Data drift (covariate shift)

What it is

The distribution of the inputs changes — P(X) moves — while the relationship between inputs and target, P(y | X), stays the same. A new marketing channel brings younger users; a new country launches; a device mix shifts; an upstream unit changes from cents to rupees.

Why it matters

It is the drift you can detect immediately, because it needs no labels: you have the production feature values the moment a request arrives. It is also the milder of the two — the learnt relationship is still correct, so the model is merely being asked about regions where it saw few training examples.

In the real world: A lending model sees many more young applicants after a marketing campaign than it was trained on.

Visual example

Data drift — P(X)

The input distribution has moved left; the model's learnt mapping is untouched.

Training distributionLast 7 days in production

Quantifying the shift

means: 39.1 -> 28.4 years, a shift of 0.9 training standard deviations

KS: statistic 0.31, p < 1e-20 — unambiguous with 40,000 live rows

tail: 22% of live traffic now falls below the training 5th percentile

effect: AUC on the under-26 segment: 0.71 against 0.80 overall at launch

Why this matters

The model is not wrong about age; it is under-trained on the ages it now mostly sees. That points at retraining or reweighting, not at re-specifying the model.

Common misconception

Wrong idea: the input distribution changed, so the model's learnt relationship is now wrong.

Correct idea: under pure covariate shift P(y | X) is unchanged — a 24-year-old still behaves the way the model says. The damage comes from having had few such rows to learn from, not from the mapping being false.

import numpy as np
from scipy import stats

ref = train_frame['age'].to_numpy()          # the training distribution
live = recent_frame['age'].to_numpy()        # last 7 days in production

print(f'train mean {ref.mean():.1f}  live mean {live.mean():.1f}')
ks = stats.ks_2samp(ref, live)
print(f'KS statistic {ks.statistic:.3f}  p {ks.pvalue:.2e}')

# Coverage: how much live traffic lands outside the training range at all?
lo, hi = np.percentile(ref, [1, 99])
print(f'outside training 1-99th pct: {((live < lo) | (live > hi)).mean():.2%}')

Covariate shift does not necessarily hurt performance, and treating every KS alert as an incident trains a team to ignore the alerts. What matters is whether the shift moves traffic into regions where the model is *uncertain or unsupported* — check the model's performance on the shifted segment before acting, and gate the alert on volume as well as distance.

Practise it: Would You Notice the Input Drifting?

Concept drift

What it is

The relationship itself changes: P(y | X) moves. The same inputs now imply a different outcome. A discount that used to drive conversion stops working; a fraud signature becomes legitimate behaviour; a symptom's diagnostic weight changes because a new variant is circulating.

Why it matters

This is the serious one, and it is invisible in input monitoring. The feature distributions can be pixel-identical to training while the model is now systematically wrong, because the only thing that moved was the mapping — and the mapping can only be observed through labels.

In the real world: Fraudsters change tactics, so the same patterns no longer mean the same thing.

Visual example

Concept drift — P(y|X)

Identical inputs, different outcomes: the learnt curve is now above the truth across the whole range.

Relationship at training timeRelationship nowWhat the model still predicts

Why input monitoring sees nothing

P(X): the discount mix is unchanged — KS 0.01, PSI 0.004, every histogram identical

P(y|X): at 30% discount: 0.58 then, 0.34 now — the mapping moved, not the inputs

symptom: mean prediction flat at 0.44 while the observed conversion rate fell to 0.31

Detection: prediction mean minus observed outcome rate, on whatever labels arrive

Why this matters

The gap between the dashed line and the red line is the model's error, and it is not visible anywhere in the input data. Only an outcome — a label, however delayed — can reveal it.

Common misconception

Wrong idea: drift monitoring on the input features will catch a degrading model.

Correct idea: input monitoring catches data drift only. Concept drift is a change in the mapping, and a model can fail completely with every feature histogram perfectly stationary.

# Concept drift shows up as a change in P(y | X) within a FIXED input bucket.
import pandas as pd

scored['bucket'] = pd.cut(scored['discount_pct'], [0, 10, 20, 30, 100])

table = scored.pivot_table(index='bucket', columns=scored['scored_at'].dt.to_period('Q'),
                           values='label', aggfunc=['mean', 'count'], observed=True)
print(table['mean'])
#  bucket      2026Q1  2026Q2  2026Q3
#  (0, 10]      0.31    0.30    0.29     <- input mix unchanged
#  (10, 20]     0.44    0.43    0.33
#  (20, 30]     0.58    0.56    0.34     <- same inputs, different outcome
#  (30, 100]    0.66    0.64    0.35

Because input monitoring cannot see it, teams with a full drift dashboard and no label pipeline are blind to the most damaging failure they can have. If labels arrive late, the interim proxy is calibration on whatever labels *do* arrive plus any fast-arriving surrogate outcome, not the feature histograms.

Label shift and the other shifts worth naming

What it is

Label shift (or prior shift) is a change in P(y) with P(X | y) unchanged — fraud doubles during a holiday while fraudulent behaviour looks the same as ever. Distinct again from covariate shift and concept drift, and it has its own fix: recalibrate the threshold or reweight the priors rather than refit the model.

Why it matters

The three shifts call for three different responses, so naming which one you have is the whole diagnosis. Covariate shift wants more data in the shifted region; concept drift wants a refit on recent data; label shift often wants nothing but a new decision threshold.

In the real world: The share of defaults rises in a downturn even if applicants look the same, which changes the right threshold.

import numpy as np

# Prior correction: adjust probabilities when only the base rate has moved.
def reprior(p, train_rate, live_rate):
    odds = p / (1 - p)
    factor = (live_rate / (1 - live_rate)) / (train_rate / (1 - train_rate))
    adjusted = odds * factor
    return adjusted / (1 + adjusted)

p = np.array([0.05, 0.20, 0.50, 0.80])
print(np.round(reprior(p, train_rate=0.02, live_rate=0.04), 3))
# [0.095 0.333 0.667 0.889] — same ranking, recalibrated levels

Ranking metrics like AUC are invariant to a pure prior shift, so a dashboard showing AUC alone will report that nothing happened while every probability the model emits is now systematically too low — and every downstream decision that compares a probability to a fixed threshold is quietly wrong. Monitor calibration, not just discrimination.

Detecting drift statistically: KS and PSI

What it is

Two standard instruments. The two-sample Kolmogorov-Smirnov test compares the largest gap between two empirical cumulative distributions and gives a p-value. The Population Stability Index bins the reference distribution and sums (live_pct - ref_pct) * ln(live_pct / ref_pct) across bins, giving a single unitless number with conventional thresholds.

Why it matters

They turn 'the histogram looks different' into a monitorable scalar with a threshold, per feature, per day. PSI in particular is widely used because its convention is stable and interpretable: under 0.1 is no meaningful shift, 0.1 to 0.25 is moderate, above 0.25 is significant.

In the real world: Banks track a population stability index monthly to know when a credit model's inputs have moved too far.

Visual example

PSI = 0.284

PSI worked bin by bin: one number, decomposable into which part of the range moved.

where the shift lives

(young up, old gone)

psi(ref, live).sum()

0.284 -> above the 0.25 action threshold

age, deciles of the training distribution

bin	train %	live %	contribution
18-24	10.0	22.0	0.0946
25-31	20.0	31.0	0.0482
32-45	40.0	34.0	0.0098
46-59	20.0	10.0	0.0693
60+	10.0	3.0	0.0843

(0.22 - 0.10) * ln(0.22 / 0.10)

0.12 * 0.788 = 0.0946

0.0946 + 0.0482 + 0.0098 + 0.0693 + 0.0843 = 0.284

Reading the convention

< 0.10: no meaningful shift — log it, do not alert

0.10-0.25: moderate — investigate, check segment performance

> 0.25: significant — treat as an incident and consider retraining

Two bins contribute 63% of the total: the alert comes with its own explanation

Why this matters

A single drift score is only actionable if you can open it up. The per-bin contributions name the affected population, which is the first thing anybody responding to the alert will ask.

Common misconception

Wrong idea: a significant drift p-value means the model needs retraining.

Correct idea: it means the inputs moved, at a sample size large enough to notice. Whether the model got worse is a separate question that needs a performance measurement on the shifted segment.

import numpy as np
from scipy import stats

def psi(ref, live, bins=10):
    edges = np.percentile(ref, np.linspace(0, 100, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    r = np.histogram(ref, bins=edges)[0] / len(ref)
    l = np.histogram(live, bins=edges)[0] / len(live)
    r, l = np.clip(r, 1e-4, None), np.clip(l, 1e-4, None)      # avoid log(0)
    return float(np.sum((l - r) * np.log(l / r)))

for col in ['age', 'spend_30d', 'tickets_7d']:
    value = psi(train_frame[col].to_numpy(), recent_frame[col].to_numpy())
    ks = stats.ks_2samp(train_frame[col], recent_frame[col])
    flag = 'SIGNIFICANT' if value > 0.25 else 'moderate' if value > 0.1 else 'stable'
    print(f'{col:<12} PSI {value:5.3f}  KS {ks.statistic:5.3f}  {flag}')

With enough rows the KS test rejects everything: at a million live rows a shift of 0.01 standard deviations is p < 0.001 and entirely irrelevant. Use an effect size — PSI, or the KS statistic itself rather than its p-value — as the alerting quantity, and reserve the p-value for small samples. Also fix the bin edges from the reference distribution once, or a re-binned PSI will drift on its own.

Monitoring the input pipeline, not just the distributions

What it is

Before any statistical drift test is meaningful, monitor the mechanical properties of the incoming data: schema (columns present, dtypes), null rate per column, unseen category rate, value range violations, duplicate rate, and row volume against its expected daily profile.

Why it matters

Most 'drift' incidents are upstream bugs. A renamed column, a units change, a join that started dropping rows, a null default that changed from NaN to 0 — each produces a dramatic distribution change with an ordinary engineering cause and a five-minute fix, and each is far more common than genuine drift.

In the real world: A broken data feed that sends zeros for income is caught by input checks before it wrecks predictions.

Visual example

Input health, 2026-09-19

The daily data-quality report: mechanical checks first, statistical drift second.

an upstream bug

(not drift)

yesterday's serving traffic vs the training reference

check	reference	yesterday	status
rows scored	118,000	121,400	ok
age null rate	1.2%	1.4%	ok
device_id null rate	3.0%	61.0%	PAGE
country unseen	0.0%	0.4%	ok
spend_30d PSI	0.00	0.09	ok

null_rate('device_id')

3% -> 61% overnight: a join broke

psi('spend_30d')

0.09 — stable, no action

One page-worthy row, with a cause that is upstream of the model entirely

What a 61% null rate does downstream

1: the imputer fills the training median for 61% of rows — they all look identical

2: a top-five feature becomes a constant, so the prediction distribution narrows sharply

3: AUC will fall next week, when the labels arrive — 7 days after the fixable cause

The null-rate check caught it in hours; the performance metric would have caught it in weeks

Why this matters

Data-quality checks are the fastest signal in the whole monitoring stack: they need no labels, no statistics and no judgement, and they catch the most common failure.

Common misconception

Wrong idea: drift detection is the monitoring problem for an ML system.

Correct idea: drift detection is the subtle part. Most production incidents are schema, null-rate or volume problems with ordinary engineering causes, and they are cheaper to detect and faster to fix.

EXPECTED = {'age': ('float64', 0, 120), 'spend_30d': ('float64', 0, 1e7),
            'country': ('object', None, None)}

def data_checks(frame, expected, train_frame, baseline_rows):
    issues = []
    for col, (dtype, lo, hi) in expected.items():
        if col not in frame:
            issues.append(f'{col}: MISSING')
            continue
        if str(frame[col].dtype) != dtype:
            issues.append(f'{col}: dtype {frame[col].dtype} != {dtype}')
        null_rate = frame[col].isna().mean()
        if null_rate > train_frame[col].isna().mean() + 0.05:
            issues.append(f'{col}: null rate {null_rate:.2%}')
        if lo is not None and frame[col].dropna().between(lo, hi).mean() < 0.999:
            issues.append(f'{col}: values outside [{lo}, {hi}]')
        if dtype == 'object':
            unseen = (~frame[col].isin(set(train_frame[col].unique()))).mean()
            if unseen > 0.01:
                issues.append(f'{col}: {unseen:.2%} unseen categories')
    if abs(len(frame) - baseline_rows) / baseline_rows > 0.3:
        issues.append(f'volume {len(frame)} vs baseline {baseline_rows}')
    return issues

A null rate jumping to 100% on one feature is often *less* visible than a subtle shift, because the pipeline fills it with a default and the model happily scores the row. Alert on null rate per feature as a first-class metric, and make the serving path refuse to score a row whose required features are missing rather than substituting a plausible-looking zero.

Practise it: A Pipeline That Does Not Leak, Would You Notice the Input Drifting?

The label-delay problem

What it is

The true outcome usually arrives long after the prediction: a chargeback settles in 60 to 90 days, a churn label is only defined after a 30-day inactivity window, a loan default takes months or years, a diagnosis is confirmed at a follow-up visit. Any metric that needs y_true is therefore computable only about the past.

Why it matters

It means real-time performance monitoring is impossible for most useful models, and accepting that shapes the whole design. You monitor label-free proxies continuously, reconcile against true performance as labels mature, and keep a prediction log complete enough to join against later.

In the real world: Whether a loan defaults is only known months later, so its accuracy can't be checked right away.

Visual example

Label maturity

Performance is knowable only behind the maturity lag; everything newer needs proxies.

the trailing 60-day window is the newest cohort whose labels are complete

input

Apr

May

Jun

Jul

Aug

Sep

output

AUC 0.79

AUC 0.77

AUC 0.74

92% labelled

41% labelled

3% labelled

What is measurable, by cohort age

Apr-Jun: fully resolved: AUC computable and trustworthy — but up to 5 months old

Jul: 92% labelled: usable with a caveat, still slightly biased toward fast resolvers

Aug-Sep: 41% and 3%: performance metrics here are noise wearing a number

AUC 0.79 -> 0.77 -> 0.74 is a real decay signal, and it is already three months stale

Why this matters

The decay visible in the resolved cohorts started three months before you could see it. That lag is exactly why label-free monitoring is not a nice-to-have but the primary signal.

Common misconception

Wrong idea: the performance dashboard shows how the model is doing.

Correct idea: it shows how the model was doing, one maturity lag ago. For a 60-day label, today's model health is a forecast from proxies, not a measurement.

import pandas as pd

preds = pd.read_parquet('prediction_log.parquet')   # request_id, p, model_version, scored_at
labels = pd.read_parquet('outcomes.parquet')        # request_id, label, resolved_at

joined = preds.merge(labels, on='request_id', how='left')
joined['days_to_label'] = (joined['resolved_at'] - joined['scored_at']).dt.days

MATURITY_DAYS = 60
cutoff = joined['scored_at'].max() - pd.Timedelta(days=MATURITY_DAYS)
mature = joined[joined['scored_at'] <= cutoff]

print(f'label coverage on mature cohort: {mature["label"].notna().mean():.1%}')
print(f'AUC on the newest mature cohort: {roc_auc_score(mature["label"], mature["p"]):.3f}')
print(f'unlabelled recent predictions: {(joined["scored_at"] > cutoff).sum():,}')

Computing performance on whatever labels have arrived so far, without waiting for maturity, is biased in a predictable direction: fast-resolving cases are systematically different from slow ones. Fraud confirms quickly and legitimate transactions simply never get flagged, so an early read on a fraud cohort looks far worse than the truth. Always compute the metric on a cohort old enough to be fully resolved, and report its age.

What to monitor while waiting for labels

What it is

Four label-free families. Input drift (per-feature PSI or KS, plus the mechanical data checks). Prediction drift: the mean, the full distribution and the rate above the decision threshold. Confidence: the share of predictions near 0.5, and distance-to-training-data measures for out-of-distribution inputs. Operational: latency, error rate, feature-fetch failures, cache hit rate.

Why it matters

Together these detect most failures within hours, long before the labels could. A prediction distribution that narrows or jumps is an especially high-value signal, because it responds to feature breakage, covariate shift and a bad deploy alike.

In the real world: Watching input changes and prediction patterns as early warning signs while true outcomes are still unknown.

Visual example

Prediction drift

The model's own output distribution collapsed toward the middle — a feature broke, not the world.

At launchYesterday

The same summary statistic, two very different days

mean: 0.33 at launch, 0.39 yesterday — a 0.06 move, easy to dismiss

sd: 0.29 -> 0.14: the model has stopped discriminating between rows

above 0.5: 31% -> 20%: the review queue shrank by a third overnight

cause: device_id nulls -> a top feature imputed to one constant for 61% of rows

Why this matters

The mean barely moved while the spread halved. A model whose outputs cluster at the base rate has lost its inputs, and this is visible the same day — with no labels at all.

Common misconception

Wrong idea: without labels there is nothing to monitor but the input features.

Correct idea: the predictions themselves are a monitorable distribution, and they aggregate every feature at once. A collapsing spread is one of the earliest and most reliable warnings available.

import numpy as np

def prediction_health(p_live, p_reference, threshold=0.5):
    return {
        'mean': float(p_live.mean()),
        'mean_ref': float(p_reference.mean()),
        'sd': float(p_live.std()),
        'sd_ref': float(p_reference.std()),
        'above_threshold': float((p_live > threshold).mean()),
        'above_threshold_ref': float((p_reference > threshold).mean()),
        'psi': psi(p_reference, p_live, bins=10),
        'near_boundary': float((np.abs(p_live - threshold) < 0.05).mean()),
    }

health = prediction_health(live_scores, launch_scores)
if health['psi'] > 0.25 or abs(health['mean'] - health['mean_ref']) > 0.05:
    alert('prediction distribution shifted', health)

The alert rate above the decision threshold is the metric with direct operational consequences and it is the one most often left unmonitored. If it doubles, the review queue doubles, and a downstream team discovers your model change before you do.

Alerting thresholds and retraining triggers

What it is

A decision table written before any alert fires: for each signal, the threshold, the sustain window, whether it pages a human or opens a ticket or triggers an automated action, and what that action is. A page is for something a person must decide now; an automatic retrain is for something a rule can handle.

Why it matters

Thresholds chosen while an incident is happening are chosen badly. Writing them in advance also forces the useful conversation — how bad is bad enough to act, and who is woken up — and prevents the two failure modes: an alert nobody acts on, and a threshold so loose the first warning is a stakeholder's email.

In the real world: Automatically alerting the team, or retraining, when drift passes a set level.

Visual example

Alert policy

Every alert needs a threshold, a window and an owner's action decided in advance.

pages: minutes matter

(fast, fixable, user-visible)

sustained window

what stops a single noisy interval paging anyone

monitoring rules, churn model v20260919

signal	threshold	sustained	action
serving error rate	> 0.5%	5 min	page on-call
required feature nulls	> 10%	15 min	page on-call
prediction mean delta	> 0.05	1 hour	page on-call
max feature PSI	> 0.25	24 hours	ticket + retrain candidate
mature cohort AUC drop	> 0.03	1 cohort	ticket + retrain candidate

PSI > 0.25 for 24h

slow signal -> ticket, never a page

Fast operational faults page; slow statistical drift opens a ticket

Choosing a sustain window

too short: one noisy 1-minute interval pages someone at 03:00 for nothing

too long: a real outage burns 30 minutes of traffic before anyone is told

rule of thumb: long enough that the metric's own noise cannot cross it, no longer

Every page must have an action. An alert with no action is a training exercise in ignoring alerts

Why this matters

The split between page and ticket is the whole policy. Operational faults are fixable in minutes and visible to users; drift is a week-scale decision that needs analysis, and paging for it destroys trust in the pager.

Common misconception

Wrong idea: more sensitive thresholds mean safer monitoring.

Correct idea: an alert that fires often and is dismissed is worse than no alert, because it teaches the team to dismiss. Sensitivity is only worth what the response to it is worth.

RULES = [
    # signal,              threshold,  sustained,   action
    ('serving_error_rate',  0.005,     '5m',        'page'),
    ('p99_latency_ms',      50,        '10m',       'page'),
    ('required_feature_null_rate', 0.10, '15m',     'page'),
    ('prediction_mean_delta', 0.05,    '1h',        'page'),
    ('feature_psi_max',     0.25,      '24h',       'ticket + schedule retrain'),
    ('mature_cohort_auc_drop', 0.03,   '1 cohort',  'ticket + schedule retrain'),
    ('mature_cohort_auc_drop', 0.07,   '1 cohort',  'page + consider rollback'),
]

def evaluate(rules, metrics):
    return [(sig, thr, act) for sig, thr, _, act in rules if metrics.get(sig, 0) > thr]

# Retraining is a trigger with a gate, never an unattended promotion.
def maybe_retrain(metrics):
    if metrics['feature_psi_max'] > 0.25 or metrics['mature_cohort_auc_drop'] > 0.03:
        candidate = train_on_recent_window()
        if beats_champion_offline(candidate) and skew_test_passes(candidate):
            return register_and_start_canary(candidate)
    return None

Never let a drift trigger promote a retrained model automatically. Drift is often caused by bad data, and retraining on bad data bakes the bug into the model — then the drift alert clears, because the new model's predictions match the broken inputs. A trigger produces a candidate; the candidate still has to pass the offline gate, the skew test and a canary.

Retraining strategies

What it is

Three approaches with different costs. Scheduled: refit on a fixed cadence regardless of drift. Triggered: refit when a drift or performance rule fires. Continual or online: update the model incrementally as data arrives, via partial_fit or an online learner.

Why it matters

The right choice follows from the decay slope measured earlier, the cost of a retraining run, how quickly labels arrive, and how much risk an unattended model update carries. Scheduled is the default for good reason: it is predictable, easy to test, and its failures are boring.

In the real world: Retraining the demand model every month, or whenever performance drops, whichever comes first.

Visual example

Retraining cadence

Three cadences, three failure modes. Pick by decay slope and label latency, not by ambition.

the sane default

(predictable and testable)

when each strategy is the right one

strategy	fits when	main risk
Scheduled (weekly)	steady decay, labels within the cadence	retrains on bad data on schedule
Triggered by drift	rare, sharp regime changes	trigger fires on an upstream bug
Continual / online	fast labels, fast decay, high volume	no stable artifact to roll back to
No retraining	stationary domain, audited model	silent decay nobody is measuring

trigger: PSI > 0.25

candidate only — never auto-promote

partial_fit(X_batch, y_batch)

needs labels in minutes, not months

Online learning is the most demanding option, not the most advanced one

What the choice actually depends on

decay slope: fraud at -0.0085 AUC/week needs weeks; credit risk at -0.0006 needs quarters

label latency: a 60-day label makes continual learning arithmetically impossible

run cost: if a refit is cheap, schedule it; triggers exist to avoid expensive refits

risk: every unattended update is a deploy with no human in the loop

Why this matters

Online learning is often chosen for sophistication and then abandoned, because it removes the one thing incident response depends on: a fixed, named, reproducible artifact you can roll back to.

Common misconception

Wrong idea: continual learning keeps a model permanently fresh, so it is the ideal to aim at.

Correct idea: it keeps the model permanently unversioned. With a fast-moving domain and minute-scale labels it is the right tool; with a 60-day label it cannot work at all.

# Scheduled, with a rolling window and an offline gate before any promotion.
def weekly_retrain(history, window_days=180, holdout_days=14):
    cutoff = history['event_ts'].max() - pd.Timedelta(days=holdout_days)
    start = cutoff - pd.Timedelta(days=window_days)
    train = history[(history['event_ts'] > start) & (history['event_ts'] <= cutoff)]
    holdout = history[history['event_ts'] > cutoff]          # strictly later, no shuffle

    candidate = clone(pipe).fit(train[FEATURES], train['label'])
    new = roc_auc_score(holdout['label'], candidate.predict_proba(holdout[FEATURES])[:, 1])
    old = roc_auc_score(holdout['label'], champion.predict_proba(holdout[FEATURES])[:, 1])
    print(f'candidate {new:.4f}  champion {old:.4f}')
    return candidate if new > old + 0.002 else None            # require a real margin

A retraining window is a real hyperparameter with a real tradeoff: too long and the model is diluted by a world that no longer exists; too short and it is fitted to noise and to seasonality. Tune it as you would any other, by backtesting several window lengths on rolling-origin splits — and always evaluate a candidate on a *strictly later* holdout, because a shuffled evaluation of a retrained model is the temporal leakage from module 22 all over again.

Feedback loops

What it is

The model's own outputs shape the data it will be trained on next. A fraud model that blocks a transaction never learns whether it was fraud; a recommender is trained on clicks from items it chose to show; a credit model only observes repayment for applicants it approved.

Why it matters

Retraining on this data amplifies whatever the model already believed. Its confident regions get more data, its uncertain regions get less, and the next model is more confident about the same things — including the mistakes. The loop is invisible in every offline metric, because the missing outcomes are missing from the evaluation set too.

In the real world: A model that only approves certain customers only ever learns about those customers, and its blind spots grow.

import numpy as np

# Keep a small randomised holdout that the model never gets to veto.
rng = np.random.default_rng(20260919)

def decide(request, model, explore_rate=0.01):
    p = float(model.predict_proba(build_features(request))[0, 1])
    if rng.random() < explore_rate:
        return {'action': 'allow', 'p': p, 'reason': 'exploration_holdout'}
    return {'action': 'block' if p > 0.5 else 'allow', 'p': p, 'reason': 'model'}

# The holdout is the only source of unbiased outcomes for high-p rows.
# It also lets you measure what blocking actually bought:
# holdout_fraud_rate(p_bucket) vs the model's predicted rate in that bucket.

A small randomised exploration holdout is the standard remedy and it has a direct, visible cost — some fraud is allowed through on purpose. That cost has to be argued for in advance, because it is the price of having any unbiased data at all, and after an incident nobody will approve it.

Closing the loop

What it is

A monitored model is a cycle, not a delivery: serve, log every prediction with its inputs and model version, watch the label-free signals continuously, reconcile against true performance as each cohort matures, and when a threshold is crossed, go back to the data — with the drifted segment already identified, which is a far better starting point than the original project had.

Why it matters

It is the shape of the whole syllabus. The loop reuses everything: the pipeline that ships, the attribution tools that explain an alert, the slicing that localises it, the experimental discipline that decides whether the replacement is genuinely better, and the canary that releases it safely.

In the real world: Feeding real outcomes back into training so the fraud model keeps up with new tactics.

Visual example

The loop, running

The last picture in the syllabus: one metric over time, an alert threshold, a drifted window, and a retrain.

Matured-cohort AUCMax feature PSI (x0.4, offset)

The full cycle, week by week

week 8: max PSI passes 0.25 -> ticket, retrain candidate scheduled. AUC still 0.781

week 10: prediction mean shifts +0.04; drifted segment localised to the under-26 cohort

week 12: matured AUC 0.719 crosses the 0.72 floor — the page the ticket predicted

week 15: candidate trained on a 180-day rolling window beats champion by +0.019 on a later holdout

week 16: 1/5/25/100% canary, guardrails clean, AUC back to 0.799

The label-free signal led the label-based one by four weeks. That lead time is the whole point

Why this matters

This is what all twenty-five modules were for: a model in production whose decay is visible before it hurts, attributable to a named segment, and repairable by a loop that has been rehearsed.

Common misconception

Wrong idea: shipping the model is the finish line, and monitoring is a dashboard somebody checks.

Correct idea: shipping is the point where the loop starts turning. The dashed drift line moves first, the solid performance line follows, and the gap between them is the time you have to act.

def monitoring_cycle(today):
    inputs = data_checks(recent_frame, EXPECTED, train_frame, baseline_rows=118_000)
    drift = {c: psi(train_frame[c].to_numpy(), recent_frame[c].to_numpy()) for c in FEATURES}
    preds = prediction_health(live_scores, launch_scores)
    mature = mature_cohort_metrics(today, maturity_days=60)

    report = {'date': str(today), 'input_issues': inputs,
              'max_psi': max(drift.values()), 'worst_feature': max(drift, key=drift.get),
              'prediction': preds, 'mature_auc': mature['auc'],
              'auc_drop': mature['auc_launch'] - mature['auc']}

    for signal, threshold, _, action in RULES:
        if report.get(signal, 0) > threshold:
            dispatch(action, signal, report)
    return report

The commonest failure of the whole module is not a bad threshold but an incomplete prediction log. If the inputs, the prediction, the model version and the timestamp were not stored together at serving time, then when the labels finally arrive there is nothing to join them to — no per-segment performance, no calibration check, no drift attribution. Log first; you cannot reconstruct it later.