Machine Learning · Module 23 of 26 · 10 concepts · about 18 min

Error Analysis & Data Leakage

Treating a metric as the start of an investigation rather than the end of one — and recognising the specific ways a dataset tells a model the answer.

At work: Finding where a model goes wrong, and catching models that only looked good because they peeked at the answer.

Reading a confusion matrix as a list of investigations

What it is

The four cells of a binary confusion matrix are four different populations: rows the model caught, rows it missed, rows it falsely flagged, and rows it correctly left alone. Each cell is a cohort you can pull, describe and compare against the others.

Why it matters

Accuracy, precision and recall are summaries *of* this table, and a summary cannot tell you that 80% of the false negatives share one product category. The table's value is as an index into the data, not as a source of numbers.

In the real world: Each group of mistakes, like good customers blocked, becomes a list of cases to review with the fraud team.

Visual example

Fraud model, 10,000 val rows

Not four numbers — four cohorts to pull and describe. The interesting ones are off the diagonal.

the 180 misses

(cost: refunded losses)

confusion_matrix(y_val, pred) at threshold 0.5

	pred: legit	pred: fraud
actual: legit	9380	120
actual: fraud	180	320

X_val[(y==1) & (pred==0)]

180 rows — what do they share?

X_val[(y==0) & (pred==1)]

120 rows — blocked good customers

Accuracy 97.0%, which is worse than always predicting legit (95.0%)

The same table, four ways

accuracy: (9380 + 320) / 10000 = 0.970 — dominated by the majority class

precision: 320 / (320 + 120) = 0.727 — of those we block, 73% were fraud

recall: 320 / (320 + 180) = 0.640 — we catch 64% of fraud

Baseline 'always legit': accuracy 0.950, recall 0.000. Accuracy alone cannot see the difference

Why this matters

The two highlighted cells have different business costs and different root causes. Collapsing them into one accuracy figure discards both the cost structure and the debugging lead.

Common misconception

Wrong idea: a high-accuracy model has few errors worth investigating.

Correct idea: with a 5% positive rate, accuracy is mostly a report on the majority class. The 180 missed frauds are the entire product problem, and they are 1.8% of the table.

from sklearn.metrics import confusion_matrix, classification_report
import numpy as np

proba = pipe.predict_proba(X_val)[:, 1]
pred = (proba >= 0.5).astype(int)
print(confusion_matrix(y_val, pred))
print(classification_report(y_val, pred, digits=3))

# The point: cells as cohorts, not counts.
fn = X_val[(y_val == 1) & (pred == 0)]
fp = X_val[(y_val == 0) & (pred == 1)]
fn['product'].value_counts(normalize=True).head(3)
fn['amount'].describe()[['mean', '50%']], X_val['amount'].describe()[['mean', '50%']]

confusion_matrix returns rows as truth and columns as prediction, so the off-diagonal cells are [0, 1] = false positive and [1, 0] = false negative. Reading it transposed swaps precision and recall — and a report claiming 94% recall when it is actually 94% precision can survive a long way through a project.

Practise it: Macro and Micro F1, From the Confusion Matrix Alone, Precision, Recall and F1 by Hand

Slicing errors by subgroup

What it is

Compute the metric separately within each level of a categorical column, each bucket of a numeric one, and each time window — then compare against the overall figure and against each slice's support.

Why it matters

An aggregate metric is an average over subpopulations that may behave nothing alike. Errors concentrated in one segment mean a specific, findable cause — a missing feature for that segment, a data pipeline that fills a column differently there, or simply too few training rows to learn it.

In the real world: A delivery-time model that's accurate overall may be badly wrong in one city, or for night-time orders.

Visual example

Slice report

Overall AUC 0.84 is an average over segments that range from 0.58 to 0.91.

read support first

(n = 41 explains nothing)

AUC by country, overall = 0.84

slice	n	positive rate	AUC
IN	4200	0.05	0.91
US	3100	0.04	0.88
BR	1900	0.11	0.58
DE	41	0.02	0.55

frame.query("country == 'BR'")

1,900 rows, AUC 0.58 — a real failure

frame.query("country == 'DE'")

41 rows — noise, not a finding

Investigate BR; ignore DE until it has enough rows to say anything

Where the BR trail leads

1: BR positive rate 0.11 vs 0.05 elsewhere — a different underlying process

2: BR rows have `device_id` null 74% of the time vs 3% elsewhere

3: A top-five feature is simply absent for this segment — a pipeline bug, not a model bug

Why this matters

The overall number gave no lead at all. One groupby produced a segment, a suspected feature and a testable cause, in that order.

Common misconception

Wrong idea: the worst slice is the one to fix.

Correct idea: the worst slice with enough support to be real is the one to fix. Rank by (gap x support), not by gap.

import pandas as pd
from sklearn.metrics import roc_auc_score

frame = X_val.copy()
frame['y'] = y_val.values
frame['p'] = pipe.predict_proba(X_val)[:, 1]

def slice_report(frame, by):
    out = frame.groupby(by).apply(
        lambda g: pd.Series({
            'n': len(g),
            'positive_rate': g['y'].mean(),
            'auc': roc_auc_score(g['y'], g['p']) if g['y'].nunique() > 1 else float('nan'),
        }),
        include_groups=False,
    )
    return out.sort_values('auc')

slice_report(frame, 'country').head()
slice_report(frame, pd.qcut(frame['amount'], 4)).head()

Small slices produce wild metrics by chance, so the worst-looking slice is often just the smallest one. Always report support beside the metric, and check whether the gap survives a bootstrap interval before declaring a subgroup problem — with 40 rows, an AUC of 0.55 and an AUC of 0.80 are not distinguishable.

Residual analysis for regression

What it is

A residual is y_true - y_pred. Plotted against the fitted value, or against any individual feature, the residuals should look like a structureless horizontal band around zero. Any visible pattern is signal the model failed to capture.

Why it matters

Each pattern names its own fix. A curve means a missing non-linear term; a widening fan means variance grows with the prediction, so the target may want a log; a step means a subgroup the model has no feature for; a drift against a date column means a time effect nobody modelled.

In the real world: Finding that a house-price model always underprices homes near new metro stations.

Visual example

Residuals vs fitted

Residuals versus fitted: a curve and a fan, not a band. Both patterns have a named fix.

ResidualsSmoothed mean residual

Two findings, two fixes

curvature: mean residual -5.5 -> +2.5 -> -2.0 across the range: add a quadratic term or use a tree

heteroscedasticity: residual sd 3 at the low end, 14 at the high end: model log(y), or use a Poisson/Tweedie loss

Overall mean residual -0.3 and correlation 0.02 — both patterns invisible in the summary statistics

Why this matters

The summary statistics passed. The plot shows two distinct, separately fixable modelling errors, each worth more than any hyperparameter tuning on this dataset.

Common misconception

Wrong idea: low RMSE means the residuals are fine.

Correct idea: RMSE is one number over the whole range. A model can hit a good RMSE while being reliably too high in the middle and wildly uncertain at the top — which matters enormously if the decisions you make live at the top.

pred = pipe.predict(X_val)
resid = y_val - pred

import numpy as np
print(resid.mean(), resid.std())                      # mean should be ~0
print(np.corrcoef(pred, resid)[0, 1])                  # correlation should be ~0

# Structure, quantified rather than eyeballed: mean residual per fitted bucket.
import pandas as pd
bucket = pd.qcut(pred, 5)
by_bucket = pd.DataFrame({'resid': resid, 'bucket': bucket}).groupby('bucket', observed=True)
by_bucket['resid'].agg(['mean', 'std', 'count'])

A residual mean of zero is guaranteed by the fitting procedure for an unregularised linear model on training data and says nothing. Judge residuals on held-out data, and judge them *conditionally* — per bucket, per feature — because the unconditional mean can be exactly zero while every bucket is badly biased in alternating directions.

Target leakage

What it is

A feature that is derived from, caused by, or a proxy for the label — and therefore would not exist, or would hold a different value, at the moment the prediction is actually needed. cancellation_reason in a churn model, total_paid in a default model, ticket_closed_at in an escalation model.

Why it matters

It is the most common cause of a model that validates brilliantly and fails completely. Nothing about the code is wrong; the dataset simply contains the answer, and every honest cross-validation scheme will faithfully report how well the model reads it.

In the real world: A churn model that uses “cancellation reason” already knows the answer, looks perfect in testing and fails live.

Visual example

Target leakage

One column recorded after the outcome. The model is not predicting churn; it is reading it.

written at cancellation

(does not exist beforehand)

corr(refund_issued, churned)

0.94 — and it is a consequence, not a cause

value at serving time

always 0 — the refund has not happened yet

training table for a churn model

customer	tenure	refund_issued	churned (label)
c-101	11	1	1
c-102	34	0	0
c-103	7	1	1
c-104	26	0	0

Offline AUC 0.982, live AUC 0.786 — the difference is one column

The timeline that gives it away

t0: prediction is needed here — refund_issued is 0 for everyone

t0 + 3d: customer cancels; the label becomes 1

t0 + 4d: refund is processed; refund_issued becomes 1

The training snapshot was taken at t0 + 30d, so it holds all three

Why this matters

Every validation split in this module is honest and every one of them reports 0.98, because train and test rows both come from a snapshot taken after the fact. No splitting strategy can detect this; only the timeline can.

Common misconception

Wrong idea: cross-validation protects against leakage.

Correct idea: cross-validation protects against overfitting. Target leakage is present in every fold equally, so a correct CV estimate of a leaky feature set is a precise measurement of a fiction.

# Symptom: one feature dominating an otherwise ordinary problem.
r = permutation_importance(pipe, X_val, y_val, n_repeats=10, random_state=0, scoring='roc_auc')
imp = pd.Series(r.importances_mean, index=X_val.columns).sort_values(ascending=False)
imp.head(3)
# refund_issued     0.371   <- one feature worth more than all the rest combined
# tenure_months     0.042
# support_tickets   0.031

# Test: does the model survive without it?
for col in ['refund_issued']:
    trimmed = pipe.fit(X_train.drop(columns=[col]), y_train)
    print(col, roc_auc_score(y_val, trimmed.predict_proba(X_val.drop(columns=[col]))[:, 1]))
# refund_issued 0.786   <- the honest score; 0.982 was the leak

The decisive question is not 'is this feature correlated with the target' — every useful feature is. It is 'was this value already recorded at prediction time'. Ask it of every column, with a timestamp, and ask the person who owns the table rather than guessing from the column name.

Train-test contamination

What it is

The same information appearing on both sides of a split: exact duplicate rows, near-duplicates (the same transaction logged twice, a re-uploaded photo, a resubmitted form), or multiple rows belonging to one entity — several visits by one patient, several sessions by one user — separated by a random split.

Why it matters

The test set stops being held out. The model can score well by recognising rather than generalising, and the more duplication the dataset has, the better the illusion. This is the single most frequent leak in datasets assembled from event logs.

In the real world: The same customers in both training and test data inflate the score the team reports.

Visual example

Group leakage

User u-7 appears on both sides: the test set is no longer unseen.

train (random 60%)

user	session	label
u-7	s-101	1
u-3	s-102	0
u-7	s-103	1

random split of a per-session table

test (random 40%)

user	session	label
u-7	s-104	1
u-9	s-105	0

GroupShuffleSplit(groups=user_id)

user	side
u-7	train only
u-3	train only
u-9	test only

What the score measured

random split: test AUC 0.93 — u-7's label was learnable from u-7's other sessions

group split: test AUC 0.74 — the honest number for a genuinely new user

The gap is entirely the model recognising users it had already been paid to memorise

Why this matters

Whether 0.93 or 0.74 is the right number depends on the deployment: scoring new users needs the group split, re-scoring existing users may genuinely be the easier problem. What is never acceptable is not knowing which one you measured.

Common misconception

Wrong idea: a random split is the neutral, unbiased default.

Correct idea: a random split assumes rows are independent. When several rows describe one entity they are not, and the split has to happen at the level of the thing that will be new in production.

# 1. Exact and near duplicates, before splitting anything.
print(df.duplicated().sum())
feature_cols = [c for c in df.columns if c != 'label']
print(df.duplicated(subset=feature_cols).sum())   # same features, possibly different label

# 2. When rows are grouped by entity, split by entity.
from sklearn.model_selection import GroupShuffleSplit, StratifiedGroupKFold

splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=0)
train_idx, test_idx = next(splitter.split(df, df['label'], groups=df['user_id']))
assert set(df.loc[train_idx, 'user_id']) & set(df.loc[test_idx, 'user_id']) == set()

cv = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(pipe, X, y, groups=df['user_id'], cv=cv, scoring='roc_auc')

A random split of a per-session dataset puts some of nearly every heavy user's sessions in both halves, and heavy users dominate the row count. The measured score is then mostly a measurement of user memorisation. The assertion in the snippet above — that the two id sets are disjoint — is worth writing literally, every time.

Practise it: A Split That Does Not Leak Across Groups

Temporal leakage

What it is

Training on data from after the prediction point: a random split of time-ordered data, a feature aggregated over a window that extends into the future, a label defined over a period the features already cover, or a target-encoding computed across the whole history.

Why it matters

Production always predicts forward. Any evaluation that lets the model see the future measures a task nobody will ever ask it to do, and time-series data almost always has enough autocorrelation for that to look like skill.

In the real world: Using information from after the loan was approved to predict whether it would default.

Visual example

TimeSeriesSplit

Each fold's training window is strictly earlier than its test block, with a gap for label maturation.

rolling-origin validation: train on the past, score the next block, step forward

input

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

output

NaN

test: Apr

test: May

test: Jun

test: Jul

test: Aug

NaN

Fold by fold

fold 1: train Jan-Mar -> test Apr

fold 2: train Feb-Apr -> test May

random split: train Jan, Mar, May, Aug -> test Feb, Apr — the model interpolates between known months

Rolling AUC 0.71; random-split AUC 0.88. Only 0.71 is achievable in production

Why this matters

Interpolating a gap between two known months is a far easier task than extrapolating past the end of the data, and it is not the task. The rolling estimate is lower because it is measuring the real one.

Common misconception

Wrong idea: shuffling is fine as long as the test rows themselves are never trained on.

Correct idea: the ordering is the constraint. A model that has seen May cannot be honestly scored on April — the world's own autocorrelation hands it most of the answer.

from sklearn.model_selection import TimeSeriesSplit

df = df.sort_values('event_ts')
tscv = TimeSeriesSplit(n_splits=4, test_size=len(df) // 6, gap=24)   # gap: leave a hole
for fold, (tr, te) in enumerate(tscv.split(df)):
    assert df.iloc[tr]['event_ts'].max() < df.iloc[te]['event_ts'].min()
    print(fold, df.iloc[tr]['event_ts'].max(), '->', df.iloc[te]['event_ts'].min())

# Backward-only feature windows: closed='left' excludes the current row.
df['spend_7d'] = (df.set_index('event_ts')
                    .groupby('user_id')['amount']
                    .rolling('7D', closed='left').sum()
                    .reset_index(level=0, drop=True))

gap matters more than it looks. If the label needs 30 days to be observed — did this customer churn within the month? — then a train fold ending the day before the test fold begins was trained on labels that were not yet knowable at that moment. The gap must be at least the label's own maturation period.

The 'too good to be true' signal

What it is

A validation score far above the plausible ceiling for the problem — near-perfect AUC on a noisy human behaviour task, an R² of 0.99 on a forecast, a classifier that beats the domain's published state of the art on the first attempt. The default explanation is leakage, not talent.

Why it matters

Treating a suspiciously good number as good news is how a leak survives to production, where it becomes a suspiciously bad number at the worst possible time. The cost of a two-hour investigation is nothing against the cost of finding out after launch.

In the real world: 99% accuracy on a hard fraud problem is a warning sign of leakage, not a reason to celebrate.

Visual example

Sanity band

Plausible ceiling versus reported score: a result above the band is a bug report, not a win.

Reported validation AUCLabel-shuffle control

Reading experiment 5

score: 0.98, twelve points above anything the literature reports for this task

control: 0.71 on shuffled labels — with no real signal, this must be 0.50

Both signals point the same way; the cause was a group split that became a random split

after fix: experiment 6: 0.79 real, 0.50 control — believable, and only +0.01 on experiment 4

Why this matters

The dashed control line is what makes the diagnosis certain rather than a hunch. A leak raises the shuffled-label score too, because the model is exploiting structure in the split rather than in the data.

Common misconception

Wrong idea: a big jump in validation score is the reward for a good idea.

Correct idea: a big jump is a hypothesis with two explanations, and leakage is the more common one. Earn the celebration by ruling it out first.

# A checklist you can actually run when a score looks too good.
print(imp.head(3))                                 # 1. is one feature carrying everything?

shuffled = pipe.fit(X_train, y_train.sample(frac=1, random_state=0).values)
print(roc_auc_score(y_val, shuffled.predict_proba(X_val)[:, 1]))   # 2. should be ~0.5

print(df.duplicated(subset=feature_cols).sum())    # 3. duplicate rows across the split?
print(df.groupby('user_id').size().describe())      # 4. many rows per entity?

for col in X_train.columns[:5]:                     # 5. drop-one-feature sensitivity
    p = clone(pipe).fit(X_train.drop(columns=[col]), y_train)
    print(col, round(roc_auc_score(y_val, p.predict_proba(X_val.drop(columns=[col]))[:, 1]), 3))

The label-shuffle test is the highest-value single check here and the most often skipped. Retrain on permuted labels: any model that still scores meaningfully above chance has a structural leak in the split or the pipeline, because there is no real signal left to find.

A leakage checklist

What it is

The concrete failures drawn together, in the order they are worth checking: preprocessing fitted before the split; duplicate or near-duplicate rows crossing the split; several rows per entity with a random split; a feature whose value is written after the prediction point; a feature aggregated over a window that includes the present or future; a target encoding or group statistic computed over all rows; a label maturation period shorter than the train/test gap.

Why it matters

Leakage is not one bug; it is a family with one symptom. A checklist converts 'the score seems too high' from an uneasy feeling into fifteen minutes of mechanical work with a definite answer.

In the real world: A standard pre-launch review every model goes through before it touches real customers.

def leak_audit(df, X, y, groups=None, time_col=None, feature_cols=None):
    report = {}
    report['exact_duplicates'] = int(df.duplicated().sum())
    report['feature_duplicates'] = int(df.duplicated(subset=feature_cols).sum())
    if groups is not None:
        report['rows_per_group_p95'] = float(df.groupby(groups).size().quantile(0.95))
    if time_col is not None:
        report['time_span'] = (str(df[time_col].min()), str(df[time_col].max()))
    corr = X.select_dtypes('number').corrwith(pd.Series(y, index=X.index)).abs()
    report['suspicious_corr'] = corr[corr > 0.9].to_dict()
    return report

leak_audit(df, X, y, groups='user_id', time_col='event_ts', feature_cols=feature_cols)

An absolute correlation above roughly 0.9 between a single feature and the target is a red flag on almost every real problem, but it is a screen and not a verdict: some legitimate features genuinely are that predictive. The screen tells you which column to take to the person who owns the table.

A debugging workflow

What it is

A fixed order of operations, so that a bad metric leads to a cause rather than to guessing: reproduce the number; check the baselines (majority class, one feature, last value); check for leakage if the number is high, for a bug if it is impossibly low; slice by subgroup and by time; pull the worst cohort and read the rows; form one hypothesis; change one thing; re-measure.

Why it matters

The alternative — tuning hyperparameters until the number moves — cannot find a mislabelled segment, a broken join, a feature that is null for one country, or a leak. Most large model improvements come from the data, and the data only gives them up to a deliberate search.

In the real world: A repeatable routine for finding out why last night's model scores dropped.

# 1. Baselines first: they bound what any improvement is worth.
from sklearn.dummy import DummyClassifier
base = DummyClassifier(strategy='prior').fit(X_train, y_train)
print('prior     ', roc_auc_score(y_val, base.predict_proba(X_val)[:, 1]))
print('one feature', roc_auc_score(y_val, -X_val['tenure_months']))
print('model     ', roc_auc_score(y_val, pipe.predict_proba(X_val)[:, 1]))

# 2. Then slice, worst-first, with support.
report = slice_report(frame, 'country')
report['gap'] = report['auc'] - 0.84
report['priority'] = report['gap'].abs() * report['n']
report.sort_values('priority', ascending=False).head(3)

Changing two things at once. A data fix and a hyperparameter change shipped together leave you unable to attribute the result, and if the pair nets out to zero you will conclude that both were worthless. One change, one measurement, recorded — which is exactly what the next module makes systematic.

Reading individual errors by hand

What it is

Sorting the validation set by loss, opening the fifty worst rows, and looking at them as records rather than as vectors — the actual text, the actual amounts, the actual timestamps, alongside the prediction and the truth.

Why it matters

It is the only technique that finds problems no metric can express: labels that are simply wrong, rows that are duplicated with conflicting labels, an encoding failure, a unit change halfway through the history, a feature that silently became null last March. These are invisible in aggregate and obvious on inspection.

In the real world: Reading 50 misclassified support tickets often reveals a labelling problem no metric would show.

import numpy as np

proba = pipe.predict_proba(X_val)[:, 1]
loss = -(y_val * np.log(proba + 1e-9) + (1 - y_val) * np.log(1 - proba + 1e-9))

worst = (X_val.assign(y=y_val.values, p=proba, loss=loss)
              .sort_values('loss', ascending=False)
              .head(50))
worst[['y', 'p', 'amount', 'country', 'device_id', 'event_ts']].head(10)

# What the reading usually turns up, made countable:
worst['device_id'].isna().mean(), X_val['device_id'].isna().mean()

Do this on validation data, and keep the final test set closed. Fifty test rows read by hand and then acted on are fifty test rows that have leaked through you into the model — the human-in-the-loop version of exactly the contamination this module is about.