Machine Learning · Module 8 of 26 · 10 concepts · about 18 min

Cross-Validation

Turning a single lucky number into an estimate with a spread — and the four ways a validation split leaks without raising anything.

At work: Getting a trustworthy estimate of how a model will do on next month's customers, not just on one lucky test set.

Why one train/test split is a noisy estimate

What it is

A single hold-out split reports the accuracy of one model on one particular sample of rows. Change the random seed and the number changes, often by several points on a few-thousand-row dataset, because the test set is a small sample and small samples have large sampling error.

Why it matters

Nearly every wrong modelling decision starts here: two models are compared on one split, one wins by 1%, and that 1% is entirely the split. Knowing the size of the noise is what tells you whether a difference is a difference.

In the real world: Two teams report 82% and 85% for the same churn model just because their random test sets differed.

Visual example

Split noise

The same model, scored on ten different random splits of the same data: the spread is the error bar nobody prints.

Hold-out accuracyMean of the ten

What the scatter says

range = 0.991 - 0.930 = 0.061 — six points of pure split luck

std ≈ 0.018, so a 1-point 'improvement' is well inside the noise

mean of all ten = 0.961 — the number worth quoting

Why this matters

Every model comparison you will ever make is a comparison of two numbers like these. If you do not know how wide each one is, you cannot say which model is better.

Common misconception

Wrong idea: the test accuracy is a property of the model.

Correct idea: it is a property of the model and the particular test rows. Resampling the split resamples the answer.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

X, y = load_breast_cancer(return_X_y=True)
scores = []
for seed in range(10):
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=seed)
    model = LogisticRegression(max_iter=5000).fit(X_tr, y_tr)
    scores.append(model.score(X_te, y_te))

np.round(scores, 3)          # the same model, ten different 'accuracies'
np.std(scores)               # ~0.02 — the width of the answer you were quoting

Reporting the best of several seeds. Trying ten splits and keeping the highest is not a better estimate, it is the maximum of ten noisy draws — biased upward by roughly the noise you just measured.

k-fold cross-validation

What it is

Split the data into k equal parts (folds). Train k separate models, each holding out one different fold as its validation set and training on the other k-1. Every row is used for validation exactly once, and for training k-1 times.

Why it matters

It buys you two things a hold-out cannot: every row contributes to the estimate, so the estimate is far less noisy, and you get k scores rather than one, which means you get a spread for free.

In the real world: Testing a lead-scoring model on five rotating slices of the data gives a reliable average and a sense of spread.

Visual example

KFold(n_splits=5)

Five rounds, five models. Each round holds out one fold and trains on the rest, so every row is validated exactly once.

One round

(one model, trained on 4 folds)

One fold of rows

(held out exactly once, down the column)

cv.split(X) — T = train, HOLD = validation

	fold 0	fold 1	fold 2	fold 3	fold 4
round 1	HOLD	T	T	T	T
round 2	T	HOLD	T	T	T
round 3	T	T	HOLD	T	T
round 4	T	T	T	HOLD	T
round 5	T	T	T	T	HOLD

5 models, 5 scores, every row validated once

With n = 1000 rows and k = 5

each fold holds 200 rows

each model trains on 800 rows and is scored on 200

total predictions scored = 5 x 200 = 1000 — the whole dataset

Why this matters

The diagonal is the whole idea: reading down a column shows a row is validated once, reading across a row shows a model never sees its own validation fold.

Common misconception

Wrong idea: k-fold trains one model on all the data and scores it five times.

Correct idea: it trains five separate models. The score you report is an average over five different fits, which is why it is stabler — and why cross_val_score hands you back no fitted model at all.

1PartitionShuffle, then cut the rows into k disjoint folds.
2RotateFor each fold in turn: fit on the other k-1, score on this one.
3AggregateReport the mean of the k scores, and their standard deviation.

from sklearn.model_selection import KFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier

cv = KFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(RandomForestClassifier(random_state=0), X, y, cv=cv,
                         scoring="accuracy")
scores                       # one score per fold: 5 numbers
scores.mean(), scores.std()  # the estimate, and its spread

Leaving shuffle=False (the default) on data that arrives sorted by the target or by group. KFold then cuts contiguous blocks, and fold 0 can be almost entirely one class. Always shuffle=True with a fixed random_state unless the row order is meaningful — and if it is meaningful, you want TimeSeriesSplit, not shuffling.

Practise it: A Pipeline That Does Not Leak, A Split That Does Not Leak Across Groups

Stratified k-fold, and class imbalance

What it is

StratifiedKFold builds folds that each preserve the overall class proportions. If the data is 8% positive, every fold is about 8% positive rather than whatever the shuffle happened to produce.

Why it matters

With a rare class, plain KFold gives folds whose positive rate varies wildly, and the fold-to-fold variance of your metric then measures the shuffle rather than the model. With a very rare class a fold can contain zero positives, at which point recall and ROC-AUC are undefined.

In the real world: Makes sure each test slice of fraud data actually contains some fraud cases, so every score means something.

Visual example

StratifiedKFold

The same 1000 rows at an 8% positive rate, folded two ways. Plain KFold lets the positive rate wander; stratifying pins it.

KFold(5, shuffle=True)

3.5% to 11.5% — a 3x swing

StratifiedKFold(5)

8.0% in every fold

positives per 200-row fold

	KFold	KFold rate	Stratified	Stratified rate
fold 0	23	11.5%	16	8.0%
fold 1	9	4.5%	16	8.0%
fold 2	19	9.5%	16	8.0%
fold 3	7	3.5%	16	8.0%
fold 4	22	11.0%	16	8.0%

80 positives, split evenly, instead of split by luck

Why fold 3 poisons the estimate

fold 3 holds 7 positives, so recall moves in steps of 1/7 = 14 points

a single extra catch changes that fold's recall by more than most model changes do

stratified: 16 positives per fold, so each step is 6 points — still coarse, but honest

Why this matters

Fold-to-fold spread is supposed to tell you how stable the model is. Unstratified folds make it tell you how uneven the shuffle was instead.

Common misconception

Wrong idea: stratification is a nicety for very imbalanced data.

Correct idea: it is how you make the k scores comparable to each other. Without it, the folds are scoring different problems.

from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
for tr, te in cv.split(X, y):
    print(y[te].mean().round(3))   # ~the same positive rate in every fold

cross_val_score(model, X, y, cv=cv, scoring="average_precision")

Assuming you have to ask for it. For a classifier, cross_val_score(model, X, y, cv=5) already uses StratifiedKFold — but the moment you pass cv=KFold(5) explicitly, or call train_test_split without stratify=y, you have silently opted out.

Practise it: Precision, Recall and F1 by Hand, Pick a Threshold for an Imbalanced Classifier

Leave-one-out cross-validation (LOOCV)

What it is

k-fold taken to its limit: k = n, so each of the n folds is a single row. Each model trains on n-1 rows and predicts one held-out point.

Why it matters

It is the least biased k-fold — each model sees almost all the data, so it estimates the performance of a model trained on the full dataset. It is also the most expensive and, counter-intuitively, often the *highest-variance* estimate.

In the real world: Useful when a medical study has only 60 patients and every one of them counts.

from sklearn.model_selection import LeaveOneOut, cross_val_score

cv = LeaveOneOut()                  # n_splits == len(X)
scores = cross_val_score(model, X, y, cv=cv)
scores.mean()                       # each score is 0.0 or 1.0 for a classifier

# For ridge and logistic regression there are closed-form shortcuts:
from sklearn.linear_model import RidgeCV
RidgeCV(alphas=[0.1, 1, 10], cv=None).fit(X, y)   # cv=None uses efficient LOO

Reaching for it as 'the most thorough option'. The n training sets differ by one row each, so the n models are almost identical and their errors are highly correlated — averaging correlated errors reduces variance far less than the fold count suggests. On top of that it costs n fits, and for a classifier each fold's score is a single 0 or 1. 5- or 10-fold is the standard choice for good reasons; LOOCV earns its keep mainly on very small datasets, or where a closed form makes it nearly free.

Practise it: A Pipeline That Does Not Leak, A Split That Does Not Leak Across Groups

Group k-fold: when rows are not independent

What it is

GroupKFold and StratifiedGroupKFold split by a group label — user id, patient id, session, device — guaranteeing that every row of a group lands in the same fold, so no group appears in both train and validation.

Why it matters

k-fold assumes rows are independent draws. If one user contributed 40 rows, plain k-fold puts 32 of them in train and 8 in validation, and the model can recognise the user rather than learn the pattern. The CV score then looks excellent and production performance collapses on users the model has never seen.

In the real world: Keeping all of one customer's transactions in the same slice, so the model can't recognise the customer instead of the fraud.

Visual example

GroupKFold

Twelve rows from four users. Plain KFold splits within a user; GroupKFold keeps each user whole.

KFold(3)

u1 is in all three folds — leak

rows, grouped by user_id

user	rows	KFold(3) fold	GroupKFold(3) fold
u1	4	0, 1, 2	0
u2	3	0, 2	1
u3	3	1, 2	2
u4	2	0, 1	0

GroupKFold(3)

one fold per user, always

The question is 'a new user', so the fold must be a new user

What the leak buys the model

u1 has 4 rows; KFold trains on 3 of them and validates on the 4th

any u1-specific quirk — device, phrasing, baseline level — is memorisable

reported AUC 0.94, AUC on genuinely new users 0.71: the gap is the leak

Why this matters

Your CV protocol should mirror the prediction you will actually be asked to make. If deployment means unseen users, then validation must mean unseen users.

Common misconception

Wrong idea: as long as no identical row is in both train and test, there is no leakage.

Correct idea: leakage is about information, not duplicate rows. Two rows from one patient share far more information than two rows from two patients.

from sklearn.model_selection import GroupKFold, cross_val_score

groups = df["user_id"].to_numpy()        # one label per row
cv = GroupKFold(n_splits=5)
cross_val_score(model, X, y, groups=groups, cv=cv)

# Sanity check that the guarantee actually holds:
for tr, te in cv.split(X, y, groups):
    assert not set(groups[tr]) & set(groups[te])

Forgetting to pass groups= through. cross_val_score(model, X, y, cv=GroupKFold(5)) raises, but cross_validate inside a Pipeline nested in a GridSearchCV will happily lose the argument if you do not thread it down — and a lost groups silently degrades to ordinary k-fold, which is exactly the leak you were guarding against.

Time-series cross-validation (walk-forward)

What it is

For ordered data, each fold trains on a prefix of the timeline and validates on the window immediately after it — never on anything earlier. TimeSeriesSplit implements the expanding-window form; a fixed-width rolling window is the other common shape.

Why it matters

Shuffled k-fold on a time series trains on the future to predict the past, which is not a mistake the deployed model will ever be allowed to make. It typically inflates the score enormously, because trend and seasonality make neighbouring timestamps near-duplicates.

In the real world: Testing a sales forecast only on months after the ones it trained on, the way it will actually be used.

Visual example

TimeSeriesSplit(n_splits=4)

Four expanding-window folds on 100 time-ordered rows. Training always ends before the test window starts.

Train spanTest spanfold 2 trainfold 2 testfold 3 trainfold 3 testfold 4 trainfold 4 test

The four folds, read off the diagram

fold 1: train [0, 20) -> test [20, 40)

fold 2: train [0, 40) -> test [40, 60)

fold 3: train [0, 60) -> test [60, 80)

fold 4: train [0, 80) -> test [80, 100)

Why this matters

Every train bar starts at 0 and stops where its test bar begins. That single visual property — no blue to the right of red, on any row — is the whole guarantee.

Common misconception

Wrong idea: time-series CV is k-fold with the folds taken in order.

Correct idea: the folds are nested prefixes, not a partition. Early rows train in every fold; the last rows train in none, and fold 1's model sees a fifth of the data that fold 4's model sees.

1Sort by timeThe row order *is* the protocol; an unsorted frame makes it meaningless.
2ExpandEach fold appends the previous test window to the training prefix.
3Gap if neededIf the label looks h steps ahead, drop h rows between train and test.

from sklearn.model_selection import TimeSeriesSplit, cross_val_score

cv = TimeSeriesSplit(n_splits=4, test_size=20)   # rows must be time-sorted
for tr, te in cv.split(X):
    print(tr.min(), tr.max(), "|", te.min(), te.max())
# 0 19 | 20 39
# 0 39 | 40 59
# 0 59 | 60 79
# 0 79 | 80 99

cross_val_score(model, X, y, cv=cv, scoring="neg_mean_absolute_error")

Two separate versions of the same mistake. First, forgetting that the *features* must also respect the cut — a rolling mean computed over the whole series before splitting has already read the future. Second, using a gap of zero when the label itself looks forward: if y is 'return over the next 7 days', the last 7 training rows overlap the test window, so pass gap=7.

Practise it: A Pipeline That Does Not Leak, A Split That Does Not Leak Across Groups

Aggregating fold scores: the mean and the spread

What it is

Cross-validation returns a vector of k scores. The headline estimate is the mean; the standard deviation across folds says how much the estimate would move if you re-drew the data. Report both, as mean ± std.

Why it matters

The spread decides whether a comparison is real. Model A at 0.842 ± 0.031 and model B at 0.851 ± 0.028 are the same model as far as this data can tell — their fold-wise differences are smaller than the fold-to-fold noise, so picking B is a coin flip dressed as a decision.

In the real world: “85% ± 1%” is a model you can rely on; “85% ± 8%” means results will swing from month to month.

Visual example

Paired comparison

Two models on the same five folds. The means differ by 0.009; the paired per-fold differences tell you whether that means anything.

Model AModel B

Paired, not pooled

A: 0.842 +/- 0.025 B: 0.851 +/- 0.024 — the error bars overlap heavily

per-fold B - A: +0.008, +0.011, +0.006, +0.012, +0.008

mean difference +0.009 with std 0.002 — B wins on every single fold

fold 2 drags both models down by ~0.04: shared difficulty, cancelled by pairing

Why this matters

Both curves have the same shape because fold difficulty dominates. Subtracting them removes that shared component, which is why a paired read finds a real 0.9-point gain that overlapping error bars hide.

Common misconception

Wrong idea: if the two mean +/- std ranges overlap, the models are indistinguishable.

Correct idea: overlap of the marginal spreads says little. The fold-to-fold variation is common to both models; compare the paired differences instead.

from sklearn.model_selection import cross_validate

res = cross_validate(model, X, y, cv=cv, scoring=["roc_auc", "average_precision"],
                     return_train_score=True, n_jobs=-1)
auc = res["test_roc_auc"]
print(f"{auc.mean():.3f} +/- {auc.std():.3f}")

# train vs test per fold is the overfitting diagnostic, for free:
res["train_roc_auc"].mean() - auc.mean()      # the generalisation gap

Comparing two models by their fold means when the folds are not the same folds. Pass the *same* CV object (or the same random_state) to both, then compare fold by fold — paired differences cancel the fold-difficulty noise and are far more sensitive than comparing two independently-drawn means.

Cross-validation inside a Pipeline

What it is

Any step that learns from data — a scaler's mean and variance, an imputer's median, a feature selector's chosen columns, an encoder's category statistics — must be fitted inside each fold, on that fold's training rows only. Wrapping the whole sequence in a Pipeline and passing the pipeline to cross_val_score is what makes that happen automatically.

Why it matters

This is the most common leak in practice, and it does not look like a leak. Scaling X once before cross-validating means every fold's training set was standardised using the mean of its own validation rows, so the score is optimistic by an amount you cannot estimate after the fact.

In the real world: Repeating the data preparation inside each test slice, so the churn estimate isn't inflated by leakage.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.feature_selection import SelectKBest, f_classif

# WRONG: the scaler and the selector have both seen every row
X_scaled = StandardScaler().fit_transform(X)
X_sel = SelectKBest(f_classif, k=10).fit_transform(X_scaled, y)
cross_val_score(model, X_sel, y, cv=cv)      # optimistic, sometimes wildly

# RIGHT: everything refits per fold
pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(),
                     SelectKBest(f_classif, k=10), model)
cross_val_score(pipe, X, y, cv=cv)

Feature selection is the worst offender by a wide margin. Choosing the 10 columns most correlated with y over the full dataset, then cross-validating the model on those 10, can produce a respectable AUC on data where y is pure noise — the selection step already fitted the target.

Practise it: A Pipeline That Does Not Leak, A Split That Does Not Leak Across Groups

Nested cross-validation

What it is

Two loops. The inner loop cross-validates over hyperparameter settings to pick the best one; the outer loop cross-validates that whole selection-and-fitting procedure to estimate how well it generalises. In scikit-learn, cross_val_score(GridSearchCV(...), X, y, cv=outer_cv).

Why it matters

GridSearchCV.best_score_ is the score of the winning configuration on the folds used to choose it — the maximum over many noisy numbers, so it is biased upward, often by a percent or more with a large grid. Nested CV separates 'which settings' from 'how good', so the reported number is honest.

In the real world: Gives an honest performance number for the credit model you'll report to the risk committee, even after tuning.

Visual example

Nested CV

The outer fold's test rows are used once, for scoring. All tuning happens inside its training rows.

Outer training rows

(re-split by the inner loop)

Outer test fold

(never seen by any tuning step)

one outer fold of a 5 x 5 nested run

	rows 0-19	rows 20-79	rows 80-100	role
outer	train	train	TEST	score the winner
inner 1	VAL	train	unseen	tune
inner 2	train	VAL	unseen	tune
inner 3	train	train*	unseen	tune
inner 4	train	train	unseen	tune

9 settings x 5 inner folds x 5 outer folds = 225 fits

The cost, and what each loop buys

inner loop answers 'which hyperparameters' — its best score is optimistic

outer loop answers 'how well does this whole procedure do' — this is the number you report

flat GridSearchCV.best_score_ = 0.883 vs nested mean = 0.861: the 2.2-point selection bias

Why this matters

Tuning is part of your model, so it has to sit inside the validation loop just like a scaler does. Nested CV is the same Pipeline discipline applied one level up.

Common misconception

Wrong idea: GridSearchCV already cross-validates, so its best score is a cross-validated estimate.

Correct idea: it is the best of many cross-validated estimates, which is a different and more optimistic quantity. Selecting on a number spends it.

from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier

inner = StratifiedKFold(5, shuffle=True, random_state=1)
outer = StratifiedKFold(5, shuffle=True, random_state=2)

search = GridSearchCV(
    RandomForestClassifier(random_state=0),
    {"max_depth": [4, 8, None], "min_samples_leaf": [1, 5, 20]},
    cv=inner, scoring="roc_auc", n_jobs=-1,
)

nested = cross_val_score(search, X, y, cv=outer, scoring="roc_auc")
nested.mean()                # honest estimate: 5 x 5 x 9 = 225 fits
search.fit(X, y).best_params_ # the settings to ship, chosen on all the data

Expecting nested CV to hand you a single best hyperparameter set. It does not, and it is not supposed to — each outer fold may select different settings, and that variation is itself informative. Nested CV estimates the *procedure's* performance; you then run the inner search once on the full dataset to get the model you actually deploy.

Practise it: A Pipeline That Does Not Leak, A Split That Does Not Leak Across Groups

Repeated k-fold, and choosing k

What it is

RepeatedStratifiedKFold runs the whole k-fold procedure several times with different shuffles and pools all the scores. k itself trades bias against cost: small k trains on less data (pessimistic bias), large k costs more fits and gives more correlated, smaller validation sets.

Why it matters

One 5-fold run gives five scores, which is a thin basis for a standard deviation. Three repeats give fifteen, at three times the cost — usually the cheapest available way to make a close comparison decidable.

In the real world: Repeating the test with different shuffles makes a small insurance dataset's score steadier before a decision is made.

from sklearn.model_selection import RepeatedStratifiedKFold

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)
scores = cross_val_score(model, X, y, cv=cv, scoring="roc_auc", n_jobs=-1)
len(scores)                   # 15
scores.mean(), scores.std()

Treating the 15 scores as 15 independent measurements and computing a standard error as std / sqrt(15). The repeats reuse the same rows, so the scores are correlated and that standard error is too small. Repeats reduce the noise in the *mean*; they do not manufacture new data.