Machine Learning · Module 7 of 26 · 10 concepts · about 18 min

Bias, Variance & Overfitting

Why error has three parts, how to tell which part is hurting you from two numbers, and which remedies act on which part.

At work: Working out why a model disappoints (too simple, or memorising its training data) before spending weeks on the wrong fix.

The bias-variance decomposition

What it is

Expected error on unseen data splits into three terms: bias squared (how wrong the average model of this family is), variance (how much the fitted model moves when the training sample changes), and irreducible noise. Capacity lowers bias and raises variance.

Why it matters

It explains why the best model is never the most flexible one available. The two terms move in opposite directions, so total error is U-shaped and the optimum is an interior point you have to locate rather than an extreme you can reach for.

In the real world: Explains why a model is off: too simple to capture seasonality (bias), or chasing noise in last month's data (variance).

Visual example

Model complexity

Total error is lowest where bias and variance trade off, not at either extreme — and the minimum is flat-bottomed.

Bias²VarianceIrreducible noiseTotal error

Total error at three complexities

complexity 2: 6.0 bias + 0.8 var + 0.8 noise = 7.6

complexity 5: 2.3 bias + 2.5 var + 0.8 noise = 5.6 <- minimum

complexity 10: 0.5 bias + 9.0 var + 0.8 noise = 10.3

Why this matters

It is the map every later diagnosis is read against: knowing which side of the minimum you are on decides whether to add capacity or remove it, and those are opposite actions.

Common misconception

Wrong idea: a better model has both lower bias and lower variance.

Correct idea: for a fixed dataset they trade off. Lowering both at once requires changing the problem — more data, better features, a better-suited model family — not turning the complexity dial.

1Bias fallsMore capacity lets the family express the true shape.
2Variance risesMore capacity also lets it chase this sample's noise.
3Their sum turnsTotal error falls, flattens, then rises: minimise the sum.

The total curve is the sum of the three below it. Both ends are equally bad and for opposite reasons, which is why one-directional tuning goes wrong.

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(0)
preds = []
for _ in range(200):                       # 200 resamples of the training data
    idx = rng.integers(0, len(X_train), len(X_train))
    m = make_pipeline(PolynomialFeatures(9), LinearRegression())
    m.fit(X_train[idx], y_train[idx])
    preds.append(m.predict(X_probe))
P = np.array(preds)
bias2 = np.mean((P.mean(axis=0) - y_probe) ** 2)   # average model, averaged error
variance = np.mean(P.var(axis=0))                  # spread across resamples

Treating bias and variance as qualities you can measure on one fitted model. Variance is defined across training samples, so estimating it needs resampling — bootstrap or repeated cross-validation. From a single fit you can only observe the symptoms.

Bias, and the symptoms of underfitting

What it is

Bias is the error that survives even with infinite data, because the model family cannot represent the true relationship. Its symptoms are high training error, validation error close behind it, and residuals with visible structure.

Why it matters

It is the failure mode nobody notices, because the numbers are unglamorous rather than alarming. A high-bias model looks 'stable' and reliable, and adding data or regularisation makes it worse rather than better.

In the real world: A straight-line sales forecast that misses every weekend spike is underfitting; more data won't fix it, better features will.

import numpy as np

train_err = np.mean((y_train - model.predict(X_train)) ** 2)
val_err = np.mean((y_val - model.predict(X_val)) ** 2)
train_err, val_err, val_err - train_err

# High and similar -> bias. The remedy is capacity, not data:
# more features, interactions, higher degree, a non-linear family, less alpha.

Responding to a high-bias model by collecting more data. More rows reduce variance and leave bias exactly where it was; six weeks of data collection then produces the same error and the same conclusion.

Variance, and the symptoms of overfitting

What it is

Variance is the part of the error that comes from the model being sensitive to which particular rows it was trained on. Its symptoms are near-zero training error, a large gap to validation error, and unstable coefficients or structure across resamples.

Why it matters

Variance is what more data, regularisation, and averaging all attack. Recognising it means you have three cheap remedies available; mistaking it for bias sends you to add capacity, which makes it worse.

In the real world: A churn model that is 99% right on training customers and 70% on new ones has memorised rather than learned.

Visual example

Variance

Five fits of the same flexible model on five resamples of the same data. Their disagreement is the variance.

True relationshipfit on resample Afit on resample Bfit on resample C

At x = 5, across the three fits

predictions 11.4, 7.6, 12.6 -> mean 10.5, truth 9.5

bias^2 at x = 5: (10.5 - 9.5)^2 = 1.0 — small

variance at x = 5: spread about the mean = 4.8 — large

Why this matters

It shows variance as a property of the *procedure*, not of one model, which is why the fix is averaging or constraining the procedure rather than inspecting the fit you happen to have.

Common misconception

Wrong idea: an overfitted model is one that is wrong.

Correct idea: it is one that is unstable. On average it may be right; the problem is that you get one draw, and the draw you got fits your sample's noise.

Averaged over many resamples these curves centre near the truth — low bias — but any single one of them is far off. That is variance, and it is what bagging averages away.

import numpy as np
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
for _ in range(5):                        # refit on bootstrap resamples
    idx = rng.integers(0, len(X_train), len(X_train))
    m = model.fit(X_train[idx], y_train[idx])
    print(np.round(m[-1].coef_[:4], 2))   # coefficients jumping = variance

cross_val_score(model, X_train, y_train, cv=5).std()   # fold-to-fold spread

Judging variance from the test score alone. A single number cannot distinguish 'this model family is unstable' from 'this split happened to be hard'. Resample or cross-validate; the spread is the measurement.

Diagnosing from train versus validation error

What it is

Two numbers decide the diagnosis. Both high and close together: bias. Train low, validation much higher: variance. Both low and close: done. Train error above the noise floor you believe in: bias, whatever the gap says.

Why it matters

It is the entire diagnostic procedure, and it costs one extra score call. It converts a vague 'the model is not good enough' into one of two named problems with disjoint remedies.

In the real world: Comparing the two errors tells the team whether to collect more data or make the model simpler, before spending weeks on either.

Visual example

Validation curve

One plot, both failure modes. Left of the minimum both curves are high; right of it only the validation curve rises.

Training errorValidation error

Three readings of the same curve

depth 2: train 0.34, val 0.36 -> gap 0.02, both high: add capacity

depth 5: train 0.16, val 0.22 -> gap 0.06, val minimal: keep

depth 20: train 0.00, val 0.38 -> gap 0.38: remove capacity

Why this matters

Every hyperparameter search is this plot for one parameter. Recognising the shape tells you whether the search range needs extending left or right — the most common cause of a search that finds nothing.

Common misconception

Wrong idea: the best hyperparameter is the one with the lowest training error.

Correct idea: training error cannot choose it — it falls monotonically, so it always votes for maximum complexity. Only the validation curve turns.

Training error only ever falls with complexity, which is exactly why it cannot be used to choose it.

from sklearn.model_selection import validation_curve
import numpy as np

depths = [1, 2, 3, 5, 8, 12, 20]
train, val = validation_curve(tree, X_train, y_train,
                              param_name="max_depth", param_range=depths, cv=5)
for d, t, v in zip(depths, train.mean(axis=1), val.mean(axis=1)):
    print(d, round(t, 3), round(v, 3), round(t - v, 3))

Reading the gap alone. A model with train 0.40 and validation 0.42 has a tiny gap and is badly underfitting; a model with train 0.99 and validation 0.94 has a bigger gap and is probably the better model. Read both levels, then the gap.

Learning curves: error against training-set size

What it is

Fit the same model on increasing fractions of the training data and plot both errors against size. Training error rises as the set grows; validation error falls. The gap at the right-hand edge, and the level both converge to, are the two readings.

Why it matters

It answers the question 'would more data help?' before you spend money collecting it. A closing gap says yes — you are variance-limited. Two curves already converged and still high says no, and points at bias instead.

In the real world: Shows whether labelling another 10,000 support tickets will actually improve the model, before paying for it.

Visual example

Learning curves

More data closes a variance gap and cannot touch bias. Two models, two verdicts, from the same plot.

Flexible model — trainFlexible model — validationRigid model — train & validation

What each shape licenses

flexible: gap 0.04 at n = 5,000 and still narrowing -> collect more data

rigid: gap 0.00 from n = 400, level 0.35 -> add capacity or features

either model at the 0.10 floor with no gap -> the problem is solved

Why this matters

It is the cheapest way to decide between the two expensive remedies — more data or a better model — and it is a decision teams routinely get wrong in the costly direction.

Common misconception

Wrong idea: more data always improves a model.

Correct idea: more data reduces variance only. Against a bias-limited model it buys nothing, which is why you plot the curves before funding the collection.

The dashed pair is already converged at 0.35 — three times the noise floor. No quantity of extra rows moves it, because the error is bias.

from sklearn.model_selection import learning_curve
import numpy as np

sizes, train, val = learning_curve(pipe, X_train, y_train, cv=5,
                                   train_sizes=np.linspace(0.1, 1.0, 8),
                                   scoring="neg_mean_squared_error")
for n, t, v in zip(sizes, -train.mean(axis=1), -val.mean(axis=1)):
    print(int(n), round(t, 3), round(v, 3))

Concluding 'more data will fix it' from a flat, high pair of curves. When the two have already met, extra rows change nothing: the limit is the model family or the features. More data reduces variance only, and there is no variance left to reduce.

Regularisation as variance reduction

What it is

Adding a penalty on parameter size shrinks the space of functions the fit will consider, so the fitted model moves less when the training sample changes. Ridge and Lasso from the Regression module are exactly this, and alpha is the dial.

Why it matters

It is the variance remedy that needs no extra data. Validation error against alpha is U-shaped for the same reason it is U-shaped against complexity: the penalty trades variance for bias, and somewhere between the two extremes is the minimum.

In the real world: Reining in a pricing model so it doesn't react to one-off promotions as if they were permanent.

Visual example

Regularisation

Validation error against alpha is U-shaped. Too little penalty is variance, too much is bias.

Training errorValidation error

The same tradeoff, in two columns

alpha 0.001: train 0.02, val 0.38 -> gap 0.36, classic variance

alpha 10: train 0.15, val 0.17 -> gap 0.02, the minimum

alpha 10,000: train 0.40, val 0.41 -> both high, bias

Why this matters

It re-reads the Ridge coefficient path from the Regression module in the language of error: the shrinkage you watched there is buying exactly this reduction in variance.

Common misconception

Wrong idea: more regularisation is safer, so when in doubt raise alpha.

Correct idea: past the minimum the penalty adds bias, and the error climbs as steeply as it did on the other side. Both directions are failures; only the curve locates the middle.

The minimum is broad: anything between alpha = 1 and alpha = 10 is equivalent here, so prefer the larger value — it is the simpler model.

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import validation_curve

alphas = np.logspace(-3, 4, 15)
train, val = validation_curve(Ridge(), X_train, y_train, param_name="alpha",
                              param_range=alphas, cv=5,
                              scoring="neg_mean_squared_error")
float(alphas[np.argmax(val.mean(axis=1))])      # the alpha with least error

Regularising a model that is underfitting. If train and validation error are both high, raising alpha raises them further — the model already lacks capacity, and the penalty removes more. Regularisation only helps on the high-variance side.

Early stopping

What it is

For an iteratively fitted model, monitor validation error each epoch and stop when it stops improving. The parameters at that point are less extreme than the fully converged ones, which is why it acts as implicit regularisation.

Why it matters

It costs nothing, needs no extra hyperparameter search, and it is the standard variance control for boosting and neural networks, where the equivalent penalty would be hard to choose directly.

In the real world: Stopping training once validation error stops improving saves compute time and gives a model that generalises better.

Visual example

Early stopping

Training loss keeps falling; validation loss turns. The turn is the stopping point, and everything after it is memorisation.

Training lossValidation loss

Stopping at 65 versus running to 200

epoch 65: train 0.34, val 0.36 -> gap 0.02

epoch 200: train 0.03, val 0.61 -> gap 0.58, validation 70% worse

the extra 135 epochs of training improved nothing that transfers

Why this matters

It is the same U-shape as complexity and as alpha, with epochs as the dial — which is the point: one tradeoff, several knobs.

Common misconception

Wrong idea: training longer cannot hurt; you can always keep the best checkpoint.

Correct idea: keeping the best checkpoint is early stopping. Without it, the model you ship is the one at the end of the red curve, which is the worst one you trained.

The patience parameter is what stops you from halting at a noisy blip: wait n_iter_no_change epochs without improvement before believing the turn.

from sklearn.linear_model import SGDRegressor
from sklearn.ensemble import HistGradientBoostingRegressor

SGDRegressor(early_stopping=True, validation_fraction=0.1,
             n_iter_no_change=10, tol=1e-4).fit(X_train, y_train)

hgb = HistGradientBoostingRegressor(early_stopping=True,
                                    n_iter_no_change=20).fit(X_train, y_train)
hgb.n_iter_       # where it actually stopped, not max_iter

Early-stopping on the test set. The stopping epoch is a hyperparameter chosen by looking at a score, so if that score comes from the test set, the test set has been fitted. Use an internal validation split, which is what validation_fraction exists for.

Irreducible error and the Bayes floor

What it is

The part of the error that comes from the target genuinely not being a function of the features: measurement noise, unobserved causes, inherent randomness. No model of any complexity, with any amount of data, goes below it.

Why it matters

It tells you when to stop. Without an estimate of the floor, a team can spend a quarter trying to push validation error from 0.11 to 0.05 when the floor is 0.10 and the model is already essentially optimal.

In the real world: Some customer behaviour is simply random; knowing that stops a team chasing accuracy the data can't deliver.

Visual example

Error budget

Three error budgets, same total of 0.22. What to do next is completely different in each.

Same headline number

(three different projects)

Decomposing a validation error of 0.22

	bias²	variance	noise	do next
Case A	0.02	0.02	0.18	stop — at the floor
Case B	0.02	0.16	0.04	more data, regularise
Case C	0.16	0.02	0.04	more capacity, better features

A single error number cannot tell you what to do; the split can

Reading each case from the curves

A: train ~ val, both near the estimated floor -> done

B: train << val, learning-curve gap still closing -> variance

C: train ~ val, both well above the floor -> bias

Why this matters

It converts the decomposition from theory into a work-allocation tool: the largest term names the next task, and the noise term names when there is no next task.

Common misconception

Wrong idea: with enough data and a big enough model, error goes to zero.

Correct idea: error goes to the noise floor. If identical feature rows have different targets, no function of those features can separate them — and no algorithm changes that.

import numpy as np

# Estimate the floor from repeated measurements of identical feature rows:
dup = frame.groupby(feature_cols)["y"].agg(["mean", "var", "size"])
repeated = dup[dup["size"] > 1]
float(np.average(repeated["var"], weights=repeated["size"]))   # ~ the noise floor

# Human agreement is the other standard proxy for the floor on labelled tasks.

Attributing the floor to the model. A validation error stuck a hair above the noise level is not an underperforming model — it is a solved problem. Diagnosing it as variance sends you to collect data that cannot help.

Capacity versus complexity, and what each symptom means

What it is

Capacity is how large a set of functions the model family can express; effective complexity is how much of that capacity the fitting procedure actually uses, after regularisation, early stopping and the data's own size. The second is what matters.

Why it matters

It explains models that seem to break the tradeoff: a heavily regularised model with millions of parameters can have modest effective complexity, so 'parameter count' is not a reading of overfitting risk.

In the real world: Choosing between a bigger model and a simpler one depends on which symptom the metrics show, not on what's fashionable.

Visual example

Diagnosis table

Two numbers, four verdicts. This table is the whole diagnostic procedure, and it fits on one line each.

If you see this

(validation easier than training)

Read train error, then the gap

	diagnosis	remedy
low / low	fine	ship it, monitor drift
low / high	variance (overfit)	more data, regularise, simplify, average
high / high	bias (underfit)	more capacity, features, interactions
high / low	impossible — a bug	check the split, check for leakage

Index reads train error / validation error

Why this matters

It is the table to consult before touching a hyperparameter, because two of the four rows call for actions in opposite directions and the fourth calls for debugging rather than modelling.

Common misconception

Wrong idea: validation error better than training error means the model generalises unusually well.

Correct idea: it usually means a bug: a leaky feature, an easier validation split, duplicated rows across the split, or dropout-style noise applied only at training time.

from sklearn.tree import DecisionTreeClassifier

big = DecisionTreeClassifier(max_depth=None).fit(X_train, y_train)
big.get_n_leaves()          # one leaf per training row, in the worst case

small = DecisionTreeClassifier(max_depth=4, min_samples_leaf=25).fit(X_train, y_train)
small.get_n_leaves()        # the same family, far less effective complexity

Comparing model families by parameter count. A 10-parameter polynomial of degree 9 and a 10,000-parameter regularised linear model do not have comparable overfitting risk; only the validation curve measures effective complexity.

Practise it: k-Means, and the Cluster That Loses All Its Points

Diagnose this curve: a worked trap

What it is

A gradient-boosted model reports training RMSE 0.04, validation RMSE 0.37, and the learning curve's gap is still wide at the full training-set size. Cross-validation gives 0.37 ± 0.02, so the split is not the cause.

Why it matters

Working one case end to end is what makes the table above usable. The reading is unambiguous — low training error, large stable gap, gap not closing yet — which is variance, and it rules out three tempting alternatives.

In the real world: Reading a training chart correctly before a launch meeting avoids shipping a model that only looked good on paper.

Visual example

Worked diagnosis

Read the two levels, then the gap. Low train, high stable validation, gap not yet closing: variance.

Train RMSEValidation RMSEAfter max_depth=3 + L2

The reasoning, step by step

train 0.04 is far below the 0.18 floor -> it is fitting noise: not bias

CV spread 0.02 << gap 0.33 -> not an unlucky split

gap still narrowing at n = 4,000 -> more data would also help

therefore: reduce capacity first, it is free; collect data second

Why this matters

It rehearses the loop this whole group has been building toward: measure two numbers, name the failure mode, pick the remedy that acts on that mode, and re-measure.

Common misconception

Wrong idea: a validation error of 0.37 against a floor of 0.18 means the model needs to be more powerful.

Correct idea: training error below the noise floor is proof it is already too powerful. The gap is variance, and every remedy for variance makes the model smaller, not bigger.

1Levels firstTrain 0.04 versus a 0.18 floor: impossible unless it is memorising.
2Then the gap0.33, stable across folds, so it is the model and not the split.
3Then the slopeStill narrowing, so data helps too — but capacity is cheaper.

Constraining capacity moved validation RMSE from 0.37 to 0.26 without a single new feature and without more data. The floor at 0.18 says there is a little further to go, and not much.

# The reading: train 0.04, val 0.37, CV spread 0.02, gap still open.
# -> variance. In order of cost, the remedies are:
model.set_params(max_depth=3)                 # 1. less capacity
model.set_params(l2_regularization=1.0)       # 2. regularise
model.set_params(early_stopping=True)         # 3. stop earlier
# 4. more data — the learning curve says it would still help here
# NOT: more features, more depth, longer training. Those add variance.

Reaching for more features when the diagnosis is variance. It is the instinctive move — 'the model does not know enough' — and it is exactly backwards: extra columns add capacity, widen the gap, and make the validation score worse while the training score improves.