Machine Learning · Module 10 of 26 · 10 concepts · about 19 min

Random Forest & Ensemble Learning

Why averaging hundreds of individually mediocre trees beats tuning one good one — and precisely which half of the error it removes.

At work: Combining many models into one steadier one: a reliable default for credit scoring, churn and demand prediction.

Why ensembling works: averaging cancels independent error

What it is

If n models each have error variance v and their errors are uncorrelated, the variance of their average is v/n. If the errors are correlated with average correlation rho, the average's variance floors out at rho * v no matter how many models you add.

Why it matters

That single formula is the design brief for every ensemble method. It says two things: add more models (which is cheap), and above all make their errors *less similar* (which is where all the actual technique lives).

In the real world: Like asking several experienced underwriters and averaging their views; individual mistakes cancel out.

Visual example

sqrt(rho + (1-rho)/n)

Ensemble error against the number of members, at three levels of error correlation. Correlation sets the floor; count only decides how fast you reach it.

rho = 0.6 (similar trees)rho = 0.2 (a real forest)rho = 0 (ideal, unreachable)

Where the floor comes from

var(mean) = rho x v + (1 - rho) x v / n

as n -> infinity the second term vanishes, leaving rho x v

rho = 0.6 -> sd floor sqrt(0.6) = 0.775: a 22% reduction, whatever n is

rho = 0.2 -> sd floor sqrt(0.2) = 0.447: a 55% reduction

Why this matters

Every mechanism in this module — bootstrapping rows, subsampling features at each split — exists to push rho down. That is what moves you from the red curve to the orange one, and it matters far more than moving right along either.

Common misconception

Wrong idea: more models in the ensemble always means less error.

Correct idea: error falls toward rho * v and then stops. Past a few hundred trees you are buying compute, not accuracy.

import numpy as np

rng = np.random.default_rng(0)
# 200 models, each unbiased with sd 1, sharing correlation rho
def ensemble_sd(n, rho):
    return np.sqrt(rho + (1 - rho) / n)

for rho in [0.0, 0.2, 0.6]:
    print(rho, [round(ensemble_sd(n, rho), 3) for n in (1, 10, 100, 1000)])
# 0.0 -> [1.0, 0.316, 0.1, 0.032]      independent: keeps shrinking
# 0.6 -> [1.0, 0.804, 0.777, 0.775]    correlated: stuck near sqrt(0.6)

Ensembling five models that are all the same algorithm on the same data with different seeds, and expecting a large gain. Their errors are ~0.95 correlated, so you get almost nothing — you paid five times the compute for a fraction of a point. Diversity is the resource being spent, not model count.

Bagging: bootstrap aggregating

What it is

Draw B bootstrap samples — each n rows sampled *with replacement* from the n training rows — fit one model on each, and average their predictions (or vote, for classification). Because sampling is with replacement, each sample contains about 63.2% of the distinct original rows, some of them several times.

Why it matters

It is the generic variance-reduction wrapper, applicable to any high-variance learner. The perturbation is entirely in the data, so it needs nothing from the base model except that it be unstable — which deep trees emphatically are.

In the real world: Training many credit models on different resamples of past loans and averaging them gives a steadier risk score.

Visual example

bootstrap=True

Three bootstrap samples of the same 6 rows. Each draws 6 with replacement, so rows repeat and roughly a third are left out.

One bootstrap sample

(6 draws, with replacement)

row counts per bootstrap sample

	row	tree 1	tree 2	tree 3
r1	r1	2	0	1
r2	r2	1	1	0
r3	r3	0	2	2
r4	r4	1	0	1
r5	r5	2	2	0
r6	r6	0	1	2

count == 0

out-of-bag for that tree

Each sample has 6 rows but only ~4 distinct rows

The 63.2% figure

P(a given row is missed by one draw) = 1 - 1/n

P(missed by all n draws) = (1 - 1/n)^n -> 1/e = 0.368

so ~63.2% of distinct rows are in-bag and ~36.8% are out-of-bag

tree 1 here: {r1 x2, r2, r4, r5 x2} — 4 distinct rows, r3 and r6 out-of-bag

Why this matters

Every tree sees a different weighted version of the dataset, so their errors decorrelate. The rows each tree *missed* are then a ready-made validation set, which is the next concept.

Common misconception

Wrong idea: a bootstrap sample is a random subset of the rows.

Correct idea: it is n draws with replacement, so it is the same size as the original with duplicates. The duplication is what reweights the fit; subsetting alone would just discard data.

1ResampleDraw n rows with replacement, B independent times.
2Fit independentlyOne deep tree per sample; the fits are embarrassingly parallel.
3AggregateAverage the probabilities (regression: the values) across all B trees.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score

bag = BaggingClassifier(
    DecisionTreeClassifier(),      # deep, unpruned: high variance on purpose
    n_estimators=200, max_samples=1.0, bootstrap=True,
    n_jobs=-1, random_state=0,
)
cross_val_score(DecisionTreeClassifier(), X, y, cv=5).mean()   # ~0.912
cross_val_score(bag, X, y, cv=5).mean()                        # ~0.956

Bagging a *pruned* tree. Pre-pruning was there to cut variance by hand; bagging removes variance for free and the pruning now only adds bias, which no amount of averaging can undo. Base learners inside a bagging ensemble should be grown deep.

Practise it: A Bootstrap Confidence Interval for a Metric

Random Forest: bagging plus feature subsampling

What it is

A random forest is bagging with one extra source of randomness: at every split, only a random subset of max_features columns is considered as candidates. The default is sqrt(n_features) for classification and 1.0 (all features) for regression.

Why it matters

Bagged trees are more correlated than you would hope, because one or two dominant features win the root split in nearly every bootstrap sample. Hiding those features from most splits forces trees to use the rest, which is what actually drives rho down — hence the large gain over plain bagging.

In the real world: A strong, low-fuss default for churn, credit and demand models on company tables.

Visual example

max_features

The root split of six trees in the same forest. max_features="sqrt" means the strongest feature is simply not offered most of the time.

5 of 30 columns

(a fresh draw at every node)

root split chosen, per tree

	candidates offered	root split	gain
tree 1	radius, texture, area, ...	worst radius	0.333
tree 2	texture, perim, smooth, ...	worst perimeter	0.291
tree 3	concave, area, symmetry, ...	worst concave pts	0.304
tree 4	radius, compact, mean area	worst radius	0.329
tree 5	smooth, texture, fractal, ...	worst texture	0.188
tree 6	area, concave, mean radius	worst area	0.298

max_features=None

would give 'worst radius' in all six rows

4 distinct root features instead of 1 — that is the decorrelation

Why the strong feature is usually absent

P(a specific feature is in a 5-of-30 draw) = 5/30 = 0.167

so ~83% of splits cannot use the best column, whatever its gain

tree 5 is genuinely worse (gain 0.188) — and that is the point

individually weaker, collectively far less correlated

Why this matters

The gain column shows the trade being made explicitly: each tree is allowed to be worse so that the trees disagree more. Bagging alone cannot buy that, because nothing stops every sample's greedy search converging on the same root.

Common misconception

Wrong idea: restricting features per split handicaps the forest.

Correct idea: it handicaps each tree, deliberately. The average improves because the errors being averaged are less alike — a worse committee member who disagrees is worth more than a good one who does not.

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=500,
    max_features="sqrt",      # ~5 of 30 columns considered per split
    min_samples_leaf=1,       # grow deep: variance is handled by averaging
    n_jobs=-1, random_state=0,
)
cross_val_score(rf, X, y, cv=5, scoring="roc_auc").mean()

rf.fit(X, y)
np.unique([t.tree_.feature[0] for t in rf.estimators_])   # many different root splits

Setting max_features=None to 'let each tree use all the information'. That turns the forest back into plain bagging, raises inter-tree correlation and usually loses accuracy. max_features is the forest's most important single hyperparameter — lower means more diverse and more biased trees, higher means stronger and more similar ones.

Out-of-bag (OOB) error

What it is

Each tree leaves out about 36.8% of the rows. For any row, average the predictions of only the trees that did not see it — that gives a held-out prediction for every training row, from a single fit, at no extra cost.

Why it matters

It is roughly equivalent to k-fold cross-validation for a fraction of the compute, which makes it the cheapest way to sweep a forest's hyperparameters or to answer 'are 200 trees enough' without a separate CV loop.

In the real world: A free, built-in test score for a random forest, handy when data is too scarce to hold a lot back.

Visual example

oob_score_

Row r3 was in-bag for trees 2 and 3 but out-of-bag for tree 1. Its OOB prediction uses only the trees that never saw it.

One row's OOB voters

(trees t1 and t4 only)

in-bag (I) / out-of-bag (O), 6 rows x 5 trees

	t1	t2	t3	t4	t5
r1	I	O	I	I	O
r2	I	I	O	O	I
r3	O	I	I	O	I
r4	I	O	I	I	I
r5	I	I	O	I	O
r6	O	O	I	I	I

rf.oob_decision_function_[2]

average of t1 and t4's probabilities

Every row gets a held-out prediction, from one fit

Why 500 trees and not 50

each tree leaves out ~36.8% of rows, independently

with 500 trees a row has ~184 OOB voters — a stable average

with 10 trees it has ~4, and P(zero OOB voters) = 0.632^10 = 0.9%

too few trees -> a noisy oob_score_ and a warning from sklearn

Why this matters

The O cells across one row are a validation set that costs nothing, because the bootstrap already created it. Reading the table down a column is training; reading across a row is validation.

Common misconception

Wrong idea: OOB error is the training error of the forest.

Correct idea: each row is scored only by trees that excluded it, so it is genuinely held out — but held out at the row level only, so it does not protect against group or time leakage.

rf = RandomForestClassifier(n_estimators=500, oob_score=True,
                            bootstrap=True, n_jobs=-1, random_state=0)
rf.fit(X, y)
rf.oob_score_                      # ~0.963, a held-out accuracy from one fit
rf.oob_decision_function_[:3]      # per-row OOB probabilities

cross_val_score(rf, X, y, cv=5).mean()   # ~0.961 — and 5x the work

Three ways it silently breaks. It requires bootstrap=True (with bootstrap=False there is no out-of-bag set and scikit-learn raises). With few trees, some rows have almost no OOB voters and the estimate is noisy — oob_score_ warns about this. And it gives no protection at all against group or time leakage: an OOB row can still be the same patient, or a later timestamp, as an in-bag row, so where GroupKFold was needed, OOB is not a substitute.

Variance reduction in practice: error against n_estimators

What it is

A forest's error falls steeply for the first few dozen trees, then flattens onto an asymptote. Adding trees never *increases* generalisation error — unlike boosting's n_estimators, this is not an overfitting knob, only a compute knob.

Why it matters

It settles the question people ask first. You pick n_estimators by watching the OOB curve flatten and then stopping, and you never need to cross-validate it, which frees the tuning budget for max_features and min_samples_leaf, where it actually matters.

In the real world: Past about 300 trees, a fraud forest stops improving; more trees only add cost and slow down scoring.

Visual example

n_estimators

OOB error against forest size, against the single tree it is built from. The curve flattens — it never turns back up.

Random forest (OOB error)One deep treeIrreducible floor (bias + noise)

Where the error went

1 tree: 0.100. 100 trees: 0.0369 — a 63% reduction, all of it variance

100 -> 400 trees: 0.0369 -> 0.0358, i.e. 0.001 for 4x the compute

the green floor at 0.0355 is bias + irreducible noise: averaging cannot touch it

note tree 1 alone (0.100) is worse than the unbagged tree (0.088): it saw only 63% of the rows

Why this matters

The gap between the red line and the green line is what bagging can win; the green line is what it can never win. That is the whole bias-variance story of this module in one picture.

Common misconception

Wrong idea: too many trees will overfit the forest.

Correct idea: adding trees only refines the average, so the curve is monotone down to an asymptote. n_estimators costs time and memory, not generalisation.

errors = []
for n in [1, 5, 10, 25, 50, 100, 200, 400]:
    rf = RandomForestClassifier(n_estimators=n, oob_score=True,
                                n_jobs=-1, random_state=0).fit(X, y)
    errors.append((n, round(1 - rf.oob_score_, 4)))
errors
# [(1, 0.1002), (5, 0.0668), (10, 0.0527), (25, 0.0422),
#  (50, 0.0387), (100, 0.0369), (200, 0.0360), (400, 0.0358)]

# warm_start avoids refitting from scratch for each point on the curve:
rf = RandomForestClassifier(warm_start=True, oob_score=True, n_jobs=-1, random_state=0)
for n in [50, 100, 200]:
    rf.set_params(n_estimators=n).fit(X, y)

Setting n_estimators=5000 because 'more is better'. It is not worse for accuracy, but it is 12x the training time, 12x the memory and 12x the prediction latency of 400 trees for roughly 0.0002 of error — and in a latency-bound service that is the whole budget.

Which half of the error bagging fixes

What it is

Averaging B unbiased models leaves the bias exactly where it was and divides the variance by up to B. So bagging is a pure variance treatment: it needs a base learner whose problem is variance, and it does nothing for one whose problem is bias.

Why it matters

It tells you when a forest cannot help. If a single deep tree and a depth-2 stump score about the same, your error is bias-dominated and a forest of 1000 trees will barely move it — that is the situation gradient boosting, in the next module, is designed for.

In the real world: A forest won't fix a churn model missing its key feature; it only steadies a model that is already flexible enough.

Visual example

Bias vs variance

The same wrapper on two base learners. Bagging pays off enormously on the deep tree and not at all on the stump.

single learner vs 200 bagged copies

base learner	problem	alone	bagged	gain
depth-1 stump	bias	0.879	0.883	+0.004
depth-4 tree	mixed	0.935	0.951	+0.016
unpruned tree	variance	0.912	0.956	+0.044
unpruned + max_features	variance	0.889	0.968	+0.079

bias-dominated

averaging cannot move it

variance-dominated

the weakest alone, the best averaged

Bagging divides variance; it leaves bias untouched

The decomposition, applied

error = bias^2 + variance + noise

bagging B models: bias^2 unchanged, variance -> rho x var + (1-rho) x var / B

stump: variance is already tiny, so there is nothing to divide

last row: individually the worst base learner, collectively the best forest

Why this matters

The last row is the counter-intuitive one worth remembering: the base learner with the lowest solo score produces the highest ensemble score, because its errors are the least correlated.

Common misconception

Wrong idea: bagging makes any model better.

Correct idea: it makes unstable models better. A stable, biased model bagged 200 times is the same biased model at 200x the cost.

from sklearn.ensemble import BaggingClassifier, RandomForestClassifier

# bagging a HIGH-variance learner: large gain
cross_val_score(DecisionTreeClassifier(), X, y, cv=5).mean()                      # 0.912
cross_val_score(BaggingClassifier(DecisionTreeClassifier(), n_estimators=200,
                                  random_state=0), X, y, cv=5).mean()             # 0.956

# bagging a HIGH-bias learner: almost nothing
stump = DecisionTreeClassifier(max_depth=1)
cross_val_score(stump, X, y, cv=5).mean()                                          # 0.879
cross_val_score(BaggingClassifier(stump, n_estimators=200,
                                  random_state=0), X, y, cv=5).mean()              # 0.883

Diagnosing underfitting and reaching for a bigger forest. More trees is the answer to a high train/validation *gap*; a model whose training score is also poor needs more capacity — deeper trees, better features, or boosting. Check the training score before you add estimators.

Feature importance in a forest: MDI vs permutation

What it is

rf.feature_importances_ averages each tree's mean-decrease-in-impurity, so it is computed on the *training* data from the fitting arithmetic. permutation_importance instead shuffles one column in held-out data and measures how much the score drops — a direct, model-agnostic measurement of reliance.

Why it matters

MDI has a systematic bias toward high-cardinality and continuous features: a column with many distinct values offers many candidate thresholds, so it wins splits by chance and accumulates impurity credit even when it carries no signal. Permutation importance does not have that bias, because it asks a question about predictions rather than about splits.

In the real world: Checking which inputs really drive a pricing forest, without being fooled by columns that simply have many distinct values.

Visual example

MDI vs permutation

A column of pure random noise, ranked by both measures. MDI gives it more credit than three real features; permutation gives it none.

rf.feature_importances_

training-time, rewards many thresholds

importance of each feature, two ways

feature	distinct values	MDI	permutation
worst radius	456	0.182	0.094
worst concave pts	442	0.151	0.081
random_noise	569	0.021	0
mean texture	479	0.018	0.004
is_screening (binary)	2	0.003	0.011

permutation_importance(rf, X_te, y_te)

held-out, measures actual reliance

569 distinct values buys 0.021 of MDI and 0.000 of real signal

Why cardinality buys MDI

a continuous column offers ~n-1 candidate thresholds at every node

the best of many random candidates has positive gain by chance alone

that chance gain is credited to the column and never corrected

the binary column offers exactly 1 threshold, so it cannot win by luck — and is under-credited

Why this matters

The two columns disagree in both directions: the noise column is over-credited and the binary column under-credited. Any ranking you show a stakeholder should be the permutation one, computed out of sample.

Common misconception

Wrong idea: feature_importances_ tells you which features matter.

Correct idea: it tells you which features won splits while fitting, which rewards high cardinality. It is a diagnostic of the fit, and the interpretability module returns to how much further this goes wrong.

from sklearn.inspection import permutation_importance

rng = np.random.default_rng(0)
X_aug = np.c_[X, rng.random(len(X))]        # a pure-noise continuous column
names = list(feature_names) + ["random_noise"]

rf = RandomForestClassifier(n_estimators=300, random_state=0).fit(X_tr, y_tr)
pd.Series(rf.feature_importances_, index=names)["random_noise"]   # ~0.021, not 0

perm = permutation_importance(rf, X_te, y_te, n_repeats=20, random_state=0)
pd.Series(perm.importances_mean, index=names)["random_noise"]     # ~0.000

Computing permutation importance on the training set. A forest that has memorised the training rows will show large drops for everything, including noise. Always permute on held-out data. And both measures mislead under strong correlation: permuting one of two duplicated columns changes nothing, because the model just reads the other one — so both score near zero and neither looks important.

Practise it: Permutation Importance, Averaged Honestly

Hard voting and soft voting

What it is

Hard voting takes the majority of the members' predicted labels. Soft voting averages their predicted probabilities and thresholds the average. VotingClassifier(voting="soft") does the latter; scikit-learn's RandomForestClassifier is itself a soft voter.

Why it matters

Soft voting almost always wins when the members produce meaningful probabilities, because it keeps the confidence information that a hard label throws away. Three members at 0.49, 0.49, 0.95 hard-vote to class 0; soft-voting gives 0.64 and picks class 1.

In the real world: Combining a tree model, a linear model and a KNN model into one fraud score that beats each of them alone.

Visual example

voting='soft'

One row, three members. Hard voting says class 0; soft voting says class 1. Soft is right here.

P(class 1) from each member

member	P(class 1)	hard vote	weight
logistic regression	0.49	0	1
random forest	0.49	0	1
SVC	0.95	1	1
— hard majority —	-	class 0	-
— soft average —	0.643	class 1	-

voting='hard'

2 votes to 1 for class 0

voting='soft'

(0.49 + 0.49 + 0.95) / 3 = 0.643

Hard voting discards how nearly the first two abstained

The same row, both rules

hard: labels [0, 0, 1] -> majority 0

soft: mean probability 0.643 > 0.5 -> class 1

two members were at 0.49 — a hair from voting the other way

with weights=[1, 2, 1] the forest's 0.49 counts twice: mean 0.5575, still class 1

Why this matters

A hard vote compresses each member's output to one bit before combining, so a near-tie and a confident call are treated identically. Soft voting is strictly more informative — provided the probabilities mean the same thing across members.

Common misconception

Wrong idea: soft voting is just a smoother version of hard voting.

Correct idea: they can disagree on the answer, as here. Soft voting is a different decision rule, not a refinement of the same one.

from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

vc = VotingClassifier(
    [("lr", LogisticRegression(max_iter=5000)),
     ("rf", RandomForestClassifier(n_estimators=300, random_state=0)),
     ("svc", SVC(probability=True, random_state=0))],
    voting="soft", weights=[1, 2, 1], n_jobs=-1,
)
cross_val_score(vc, X, y, cv=5, scoring="roc_auc").mean()

Soft-voting models whose probabilities are on different scales. An SVM's probability=True output and a deep forest's leaf fractions are both 'probabilities' and neither is calibrated, so the average is dominated by whichever member is most overconfident. Wrap the members in CalibratedClassifierCV first, or use hard voting, or stack — a meta-model learns the weighting instead of you guessing it.

Class weights and imbalance in a forest

What it is

class_weight="balanced" reweights each class inversely to its frequency in the impurity calculation; class_weight="balanced_subsample" recomputes those weights per bootstrap sample. Neither changes the data — they change what counts as a good split.

Why it matters

On imbalanced data an unweighted forest can reach high accuracy by predicting the majority class almost everywhere, because impurity is already near zero at most nodes. Weighting makes a minority-class error expensive enough to be worth splitting on.

In the real world: Telling the forest that a missed default costs ten times more than a false alarm.

rf = RandomForestClassifier(n_estimators=400, class_weight="balanced_subsample",
                            min_samples_leaf=5, n_jobs=-1, random_state=0)
cross_val_score(rf, X, y, cv=cv, scoring="average_precision").mean()

# and evaluate on a metric that notices the minority class:
#   average_precision / recall — never plain accuracy

Tuning class weights while still scoring with accuracy. Weighting deliberately trades majority-class accuracy for minority-class recall, so the metric you are optimising will get *worse* and you will conclude it did not work. Switch to average_precision, recall or a cost-weighted score first, then tune.

Extra Trees, and the rest of the ensemble family

What it is

ExtraTreesClassifier pushes randomisation one step further: thresholds are drawn at random rather than optimised, and by default it does not bootstrap. Beyond bagging there is boosting (sequential, the next module) and stacking (StackingClassifier, where a meta-model learns how to combine members' out-of-fold predictions).

Why it matters

They occupy different points on the same bias-variance dial. Extra Trees trades a little more bias for much less variance and much faster fitting; stacking learns the combination rule instead of fixing it as an average, which is usually the last percent available on a tabular problem.

In the real world: A faster variant for large datasets when a nightly job has to train quickly.

from sklearn.ensemble import ExtraTreesClassifier, StackingClassifier

et = ExtraTreesClassifier(n_estimators=500, n_jobs=-1, random_state=0)
cross_val_score(et, X, y, cv=5, scoring="roc_auc").mean()     # often ~= RF, much faster

stack = StackingClassifier(
    [("rf", RandomForestClassifier(n_estimators=300, random_state=0)),
     ("et", ExtraTreesClassifier(n_estimators=300, random_state=0))],
    final_estimator=LogisticRegression(max_iter=5000),
    cv=5,                    # out-of-fold predictions: the meta-model never sees in-fold ones
    n_jobs=-1,
)

Building a stack whose meta-model is trained on in-sample member predictions. A member that memorised the training set hands the meta-model near-perfect features, so the meta-model learns to trust it completely and the stack collapses in production. StackingClassifier's cv= argument exists precisely to generate out-of-fold predictions instead — never bypass it.