Machine Learning · Module 11 of 26 · 10 concepts · about 19 min

Gradient Boosting

Fitting trees to the errors of the trees before them — the strongest tabular model there is, and the easiest to overfit by leaving it running.

At work: The workhorse behind many production models on business tables: fraud, credit risk, pricing and ad click prediction.

Boosting vs bagging

What it is

Bagging fits B independent models on perturbed copies of the data and averages them. Boosting fits models one at a time, each on a version of the problem reweighted or re-targeted by the errors of the current ensemble, and *adds* them up with a small multiplier.

Why it matters

The two reduce different error components, so they want opposite base learners. Bagging wants deep unpruned trees, because it removes variance and needs low bias to start from. Boosting wants stumps or depth-3 trees, because it removes bias and would amplify variance if each member had much.

In the real world: Boosting builds models one after another, each fixing the last one's mistakes: why it often wins on fraud and pricing data.

Visual example

Bagging vs boosting

The same base learner, used two ways. Read the depth wanted and n_estimators risk rows together.

two ensembles of trees

property	Random Forest	Gradient Boosting
fits	parallel, independent	sequential, on residuals
attacks	variance	bias
depth wanted	deep (None)	shallow (3-6)
combines by	averaging	weighted sum
more estimators	safe, plateaus	overfits eventually

n_estimators=5000

slow but harmless

n_estimators=5000

almost certainly overfit

Opposite base learners, opposite stopping rules

Why the depths differ

bagging: bias^2 survives averaging, so start with a near-zero-bias (deep) tree

boosting: each round reduces bias, so a low-variance (shallow) tree is the right unit

a deep tree inside a GBM zeroes the residuals in one round — nothing left to boost

a stump inside a bagging ensemble has no variance to average away

Why this matters

Almost every gradient-boosting mistake is a random-forest reflex applied in the wrong place: growing deep, adding estimators freely, and not watching a validation curve.

Common misconception

Wrong idea: boosting is bagging with the trees added instead of averaged.

Correct idea: boosting changes what each tree is fitted to. Every tree after the first is trained on a target the earlier trees created.

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

# deep trees, averaged in parallel
RandomForestClassifier(n_estimators=300, max_depth=None, random_state=0)

# shallow trees, summed in sequence
GradientBoostingClassifier(n_estimators=300, max_depth=3, learning_rate=0.1,
                           random_state=0)

# both ~0.96-0.98 here; their failure modes are completely different

Carrying forest habits over. max_depth=None in a GBM is not 'letting it learn' — each tree then fits the residuals almost exactly, the next tree has nothing left to correct, and the ensemble overfits within a handful of rounds. Depth 3 to 6 is the working range, and n_jobs cannot parallelise across rounds because round m+1 needs round m's output.

AdaBoost: the reweighting idea

What it is

The original boosting algorithm. Every row starts with equal weight; after each weak learner is fitted, misclassified rows have their weights increased and correct ones decreased, so the next learner concentrates where the ensemble is currently wrong. Each learner also gets a vote weight alpha proportional to how well it did.

Why it matters

It is the cleanest statement of the boosting intuition — 'focus the next model on what the current one misses' — and it makes gradient boosting's generalisation legible: AdaBoost is gradient boosting with the exponential loss.

In the real world: Focusing each new model on the transactions the previous ones got wrong.

Visual example

AdaBoost weights

Five rows through three rounds. Row 3 keeps being wrong, so its weight compounds — which is the strength and the failure mode at once.

Persistently wrong

(wrong in all 3 rounds)

Uniform start

(1/n each)

sample weight after each round

	start	after 1	after 2	after 3
r1	0.2	0.125	0.087	0.062
r2	0.2	0.125	0.087	0.062
r3	0.2	0.375	0.478	0.549
r4	0.2	0.125	0.174	0.203
r5	0.2	0.25	0.174	0.124

estimator_weights_

alpha = 0.5 ln((1-err)/err)

One row holds 55% of the objective after 3 rounds

Round 1, in full

stump 1 misclassifies r3 and r5: weighted error = 0.2 + 0.2 = 0.4

alpha = 0.5 x ln(0.6 / 0.4) = 0.203

wrong rows scale by e^alpha = 1.225, right rows by e^-alpha = 0.816

renormalise to sum 1 -> r3 and r5 rise to 0.25, the rest fall to 0.167

Why this matters

The weight column is the 'problem' each successive learner is handed. Boosting never changes the rows, only how much each one counts.

Common misconception

Wrong idea: AdaBoost throws away the rows the ensemble already gets right.

Correct idea: it down-weights them. They still constrain the fit; they just stop dominating it. And a mislabelled row is indistinguishable from a hard one, which is why noise hurts so much.

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier

ada = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth=1),   # the classic weak learner: a stump
    n_estimators=200, learning_rate=0.5, random_state=0,
)
ada.fit(X_train, y_train)
ada.estimator_weights_[:5].round(3)   # each stump's alpha
ada.estimator_errors_[:5].round(3)    # its weighted error rate

Using AdaBoost on noisy or mislabelled data. A permanently misclassified row has its weight escalated round after round until it dominates the objective, so the ensemble contorts itself around individual label errors. Gradient boosting with a robust loss (loss="huber" for regression) degrades far more gracefully.

Gradient boosting: fitting the residuals

What it is

Start with a constant prediction (the target mean, or the base log-odds). At each round, compute the negative gradient of the loss at the current predictions — for squared error that is exactly the residual y - F(x) — fit a shallow regression tree to *those* values, and add it to the running prediction scaled by the learning rate.

Why it matters

It generalises boosting to any differentiable loss, which is why one algorithm covers regression, classification, ranking, Poisson counts and quantiles. It also explains the name: each tree is a step in function space along the gradient of the loss.

In the real world: Each new tree corrects the leftover error in a sales forecast, step by step.

Visual example

Training loss

Training loss against boosting round. The steep early drop is bias being removed; the long flat tail is diminishing returns.

Round 1, arithmetic

F0(x) = mean(y) = 4.20 for every row; MSE = var(y) = 0.95

one row has y = 5.60, so its residual is 5.60 - 4.20 = +1.40

tree 1's leaf for that row predicts +1.18, the mean residual of its leaf

F1(x) = 4.20 + 0.1 x 1.18 = 4.318 — a tenth of the correction, on purpose

Why this matters

This curve only ever goes down: it is training loss, and boosting is minimising it by construction. That is exactly why it cannot be used to decide when to stop — the next two concepts are about the curve you must watch instead.

Common misconception

Wrong idea: a flattening training loss means the model has converged and is done.

Correct idea: training loss falls forever, toward zero. Flattening tells you about the fit to these rows and nothing about generalisation.

1InitialiseF0(x) is a constant: the mean, or the base log-odds.
2Fit the gradientA shallow regression tree on the current negative gradients.
3Take a small stepF <- F + learning_rate * tree, then repeat.

import numpy as np
from sklearn.tree import DecisionTreeRegressor

# gradient boosting for squared error, written out
F = np.full(len(y), y.mean())         # round 0: the constant model
lr, trees = 0.1, []
for m in range(200):
    residual = y - F                   # the negative gradient of 0.5*(y-F)^2
    t = DecisionTreeRegressor(max_depth=3).fit(X, residual)
    F += lr * t.predict(X)
    trees.append(t)

# identical in spirit to:
from sklearn.ensemble import GradientBoostingRegressor
GradientBoostingRegressor(n_estimators=200, learning_rate=0.1, max_depth=3).fit(X, y)

Assuming the trees predict the target. They predict *corrections*, so an individual tree's output is meaningless on its own and is often negative even when y is strictly positive. Only the running sum F(x) — plus, for classification, a sigmoid over it — is a prediction.

Learning rate and shrinkage

What it is

The learning rate (learning_rate, also called shrinkage or eta) multiplies every tree's contribution before it is added. It trades directly against n_estimators: halving the rate roughly doubles the rounds needed to reach the same training loss.

Why it matters

Small steps generalise better. A large step commits hard to one tree's view of the residuals, including their noise; many small steps let later trees moderate earlier ones, which is empirically worth a real amount of accuracy. 0.05–0.1 with early stopping is the standard recipe.

In the real world: Smaller, careful steps usually give a more accurate credit model, at the cost of longer training.

Visual example

learning_rate

Validation loss against round, at four learning rates. The lowest curve belongs to the slowest rate — it just takes longer to get there.

lr = 1.0lr = 0.5lr = 0.1lr = 0.01

Rate against rounds

lr 1.0: best 0.190 at round ~25, then climbs steadily

lr 0.5: best 0.152 at round ~60

lr 0.1: best 0.118 at round ~200 — the lowest of the four

lr 0.01: 0.131 at round 500 and falling; it needs ~2000 rounds

note lr x best_round ~ 20-25 for all four: roughly a constant step budget

Why this matters

All four curves start at the same place and end at very different floors. The rate does not just change how fast you descend — it changes how low you can get, because a big step cannot be partly undone by later trees.

Common misconception

Wrong idea: the learning rate only controls training speed.

Correct idea: it is a regulariser. A high rate reaches a worse optimum however many rounds you give it, and a low rate is strictly better provided you can afford the rounds.

for lr in [0.01, 0.1, 0.5, 1.0]:
    gb = GradientBoostingClassifier(learning_rate=lr, n_estimators=500,
                                    max_depth=3, random_state=0)
    print(lr, round(cross_val_score(gb, X, y, cv=5, scoring="roc_auc").mean(), 4))
# 0.01 -> 0.9903   0.1 -> 0.9931   0.5 -> 0.9874   1.0 -> 0.9801

Tuning learning_rate and n_estimators independently in a grid search. They are not independent — they are close to a single 'total step budget', so the grid wastes most of its cells on (tiny rate, few rounds) which underfits and (large rate, many rounds) which overfits. Fix the rate low, then let early stopping choose the round count.

n_estimators as the overfitting knob, and early stopping

What it is

Unlike a forest, a GBM's validation error is U-shaped in the number of rounds: it falls, reaches a minimum, and then rises as later trees start fitting noise in the residuals. Early stopping monitors a validation set and halts when it has not improved for n_iter_no_change rounds.

Why it matters

It is the single most important operational fact about boosting. Leave a GBM running and it will overfit; the round count is not a budget you set, it is a parameter you *learn* from a validation curve.

In the real world: Stopping the boosting once validation stops improving prevents a click model from memorising last week's traffic.

Visual example

Early stopping

Train and validation loss against round. Training loss falls forever; validation loss turns upward at round 210.

Training lossValidation loss

What patience buys

round 210: train 0.058, val 0.124 — the minimum

round 800: train 0.006, val 0.211 — 70% better fit, 70% worse generalisation

n_iter_no_change=30 means: stop once 30 straight rounds fail to beat the best

so it actually halts near round 240 and keeps the round-210 model

Why this matters

The two curves diverging monotonically after round 210 is the whole reason n_estimators cannot be set by looking at training output. This is the same diagnostic as the tree-depth curve in module 9, with rounds in place of depth.

Common misconception

Wrong idea: as long as training loss is still falling, more rounds help.

Correct idea: training loss falls to zero regardless. Only the validation curve knows where the model stopped learning the signal and started learning the residual noise.

1Hold outCarve a validation slice from the training data, not the test set.
2WatchScore the ensemble after every round; remember the best.
3Stop with patienceHalt after n_iter_no_change rounds of no improvement, and keep the best.

from sklearn.ensemble import HistGradientBoostingClassifier

gb = HistGradientBoostingClassifier(
    learning_rate=0.05, max_iter=2000,
    early_stopping=True, validation_fraction=0.15,
    n_iter_no_change=30, random_state=0,
)
gb.fit(X_train, y_train)
gb.n_iter_                     # e.g. 214 — where it actually stopped

# With the classic API, the staged curve is available directly:
gbc = GradientBoostingClassifier(n_estimators=800, learning_rate=0.05).fit(X_tr, y_tr)
val = [log_loss(y_val, p) for p in gbc.staged_predict_proba(X_val)]
int(np.argmin(val)) + 1        # the best round

Early stopping against the same validation set you later report on. The stopping round was chosen on those rows, so it is a fitted parameter and the reported score is optimistic. Use an inner validation split (validation_fraction carves one out of the training data) and keep the outer test set untouched — the nested-CV discipline from the Cross-Validation module, applied to the round count.

Regularising a GBM: the other knobs

What it is

Beyond the rate and the round count: max_depth / max_leaf_nodes cap each tree's interaction order, min_samples_leaf (min_child_weight in XGBoost) refuses leaves built on too few rows, subsample < 1 fits each tree on a random fraction of rows (stochastic gradient boosting), colsample_bytree samples columns, and reg_lambda / reg_alpha place L2 / L1 penalties directly on the leaf output values.

Why it matters

A GBM has many more ways to overfit than a forest and correspondingly more ways to be held back. Knowing which knob addresses which failure — too-complex trees, too-confident leaves, too-similar trees — is what turns tuning from a random search into a diagnosis.

In the real world: Limiting tree depth and sampling rows keeps a churn model from latching onto one-off quirks.

Visual example

Diagnosis

Which knob to reach for, indexed by the symptom rather than by the API.

symptom -> knob

symptom	first knob	direction	why
large train/val gap	max_depth	lower	fewer interactions to memorise
val loss spiky per round	learning_rate	lower	smaller, steadier steps
overconfident probabilities	reg_lambda	raise	shrinks leaf output values
train loss also poor	max_depth	raise	bias, not variance — add capacity
trees nearly identical	subsample	lower	row noise decorrelates them

underfitting

the one row where you add capacity

Diagnose from the train/validation pair, then turn one knob

Reading the gap first

train 0.02 / val 0.21 -> variance: lower depth, raise min_child_weight, subsample

train 0.19 / val 0.21 -> bias: raise depth, more rounds, better features

train 0.02 / val 0.12 and val still falling -> just keep boosting

Why this matters

Every knob here is either a capacity limit or a noise injection, and the train/validation gap tells you which family you need before you touch anything.

Common misconception

Wrong idea: regularising a GBM means turning all the regularisers up.

Correct idea: it means turning up the one that matches the symptom. Raising every knob at once usually produces an underfit model that is also slow to tune.

import xgboost as xgb

model = xgb.XGBClassifier(
    n_estimators=2000, learning_rate=0.05,
    max_depth=4,              # interaction order: depth d ~ d-way interactions
    min_child_weight=5,       # no leaf built on fewer than ~5 effective rows
    subsample=0.8,            # 80% of rows per tree: noise that decorrelates
    colsample_bytree=0.8,     # 80% of columns per tree
    reg_lambda=1.0,           # L2 on the leaf weights
    early_stopping_rounds=50, eval_metric="logloss",
)
model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
model.best_iteration

Grid-searching all seven knobs at once. That is a combinatorial explosion in which most cells are obviously bad; the effective practice is a low fixed rate, early stopping for the rounds, then a modest random search over max_depth, min_child_weight and subsample only — which is typically three knobs and a few dozen fits rather than thousands.

XGBoost, LightGBM and CatBoost: what actually changed

What it is

All three implement the same algorithm faster and with better defaults. The main practical changes: histogram-based splitting (bin each feature into ~256 buckets once, so a split search is over bins rather than sorted values), leaf-wise growth in LightGBM (expand the single highest-gain leaf anywhere in the tree, rather than level by level), native categorical handling, native missing-value routing, and second-order (Newton) steps using the loss's Hessian as well as its gradient.

Why it matters

Histogram binning alone turns an O(n log n) split search per feature into O(n + bins), which is what made gradient boosting practical on millions of rows. Everything else is refinement around that.

In the real world: Libraries used in production at banks, e-commerce and ad companies, chosen for speed and how they handle categories.

Visual example

Tree growth

Same budget of 8 leaves, two growth policies. Leaf-wise spends them all on the highest-gain branch.

level-wise: fill each depth before the next

	leaves added	total gain	shape
depth 1	2	0.333	balanced
depth 2	2	0.121	balanced
depth 3	4	0.064	balanced

level-wise (XGBoost default) vs leaf-wise (LightGBM)

leaf-wise: always split the best leaf anywhere

	leaves added	total gain	shape
split 1-2	2	0.333	unbalanced
split 3-5	3	0.171	unbalanced
split 6-7	2	0.098	unbalanced
depth	-	0.602	depth 6

Same 8 leaves, different totals

level-wise total gain = 0.333 + 0.121 + 0.064 = 0.518, max depth 3

leaf-wise total gain = 0.602, but one branch reached depth 6

more gain per leaf, and more chance to memorise a small pocket of rows

hence min_child_samples: it stops the deep branch from reaching 3-row leaves

Why this matters

Leaf-wise growth is strictly better at reducing training loss per leaf and strictly more prone to overfitting, which is exactly the trade its extra hyperparameters exist to manage.

Common misconception

Wrong idea: LightGBM is faster than XGBoost because it is better written.

Correct idea: the big wins are algorithmic: histogram binning of the features, and a growth policy that spends leaves where the gain is. XGBoost's tree_method='hist' gets most of the first one.

import lightgbm as lgb

model = lgb.LGBMClassifier(
    n_estimators=3000, learning_rate=0.03,
    num_leaves=31,            # leaf-wise: control leaves, not depth
    min_child_samples=20,     # essential with leaf-wise growth
    max_bin=255,              # the histogram resolution
)
model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)],
          callbacks=[lgb.early_stopping(50), lgb.log_evaluation(0)])

# In-library, no extra dependency:
from sklearn.ensemble import HistGradientBoostingClassifier   # same histogram idea

Carrying max_depth reasoning into LightGBM. Leaf-wise growth means num_leaves=1024 can produce a very deep, very unbalanced tree that memorises small pockets of rows — num_leaves and min_child_samples are the controls there, and max_depth is a secondary guard. Conversely, setting num_leaves on a level-wise learner does nothing useful.

When to choose boosting over a forest

What it is

Boosting usually wins on accuracy for tabular supervised learning, by anywhere from nothing to several points. A forest usually wins on time-to-a-decent-model, robustness to bad hyperparameters, and interpretability of its validation story.

Why it matters

The cost of boosting is real and is mostly human: it has more knobs, it is genuinely sensitive to them, it needs a validation set wired in for early stopping, and it trains sequentially so you cannot buy your way out with cores. A forest is close to hyperparameter-free.

In the real world: When squeezing out the last percent of accuracy on a large table pays off, as in ad click-through or fraud.

Visual example

Data-size crossover

Held-out AUC against training-set size for both families. Boosting's edge appears as the data grows and is negative at the small end.

Random forestTuned gradient boosting

The crossover, read off

100 rows: RF 0.758, GBM 0.731 — boosting loses by 2.7 points

~1.2k rows: the two curves meet

1M rows: RF 0.894, GBM 0.936 — boosting wins by 4.2 points

the forest curve also flattens sooner: it runs out of variance to remove

Why this matters

Both curves rise with data, but boosting's rises faster and for longer, because extra rows let it keep taking small bias-reducing steps. The forest plateaus once averaging has removed the variance it can remove.

Common misconception

Wrong idea: gradient boosting is simply the better tabular model.

Correct idea: it is the better asymptotic model. On a few hundred noisy rows a forest often wins outright, and it always wins on effort per point of AUC.

# A sensible default pair to run first on any tabular problem:
baseline = RandomForestClassifier(n_estimators=500, n_jobs=-1, random_state=0)

strong = HistGradientBoostingClassifier(
    learning_rate=0.05, max_iter=2000, early_stopping=True,
    validation_fraction=0.15, n_iter_no_change=30, random_state=0,
)

for name, m in [("rf", baseline), ("gbm", strong)]:
    s = cross_val_score(m, X, y, cv=cv, scoring="roc_auc", n_jobs=-1)
    print(name, f"{s.mean():.4f} +/- {s.std():.4f}")

Reaching for boosting on a small, noisy dataset. With a few hundred rows the validation split used for early stopping is itself noisy, the tuning has almost no signal to work with, and a forest's OOB estimate is both cheaper and more stable. Boosting's advantage grows with data size.

Boosting for regression: choosing a loss

What it is

The loss is a first-class hyperparameter, because the algorithm only ever needs its gradient. squared_error chases the conditional mean and is sensitive to outliers; absolute_error chases the median; huber is quadratic near zero and linear in the tails; quantile with alpha=0.9 fits the 90th percentile; poisson suits non-negative counts.

Why it matters

It lets you state what a wrong prediction actually costs. If under-forecasting demand costs three times what over-forecasting does, a quantile loss encodes that directly — which no amount of tuning a squared-error model will replicate.

In the real world: A demand forecast where under-stocking hurts more than over-stocking uses a loss that reflects that cost.

from sklearn.ensemble import GradientBoostingRegressor

# a prediction interval from two quantile models
lo = GradientBoostingRegressor(loss="quantile", alpha=0.05, n_estimators=400).fit(X, y)
hi = GradientBoostingRegressor(loss="quantile", alpha=0.95, n_estimators=400).fit(X, y)
np.c_[lo.predict(X_new), hi.predict(X_new)]     # a 90% interval, per row

# robust to a few wild targets
GradientBoostingRegressor(loss="huber", alpha=0.9, n_estimators=400)

Training with one loss and selecting the model with another. If you fit quantile(alpha=0.9) but early-stop on MSE, you stop where the *mean* fit was best, which is not where the 90th-percentile fit was best. The training loss, the early-stopping metric and the reported metric should all be the one you actually care about.

Interpreting a boosted model

What it is

A GBM is a sum of hundreds of trees, so its rules cannot be read the way a single tree's can. The usable tools are permutation importance (as in the previous module), partial dependence / ICE plots for the shape of one feature's effect, and SHAP values for a per-prediction additive attribution.

Why it matters

Boosting's accuracy advantage is often spent arguing about trust. Knowing that a per-prediction explanation is available — and roughly what it does and does not claim — is what makes the model shippable in a regulated or reviewed setting.

In the real world: Explaining to a customer why their insurance quote came out higher, feature by feature.

from sklearn.inspection import PartialDependenceDisplay, permutation_importance

perm = permutation_importance(model, X_val, y_val, n_repeats=20, random_state=0)
order = np.argsort(perm.importances_mean)[::-1][:4]

PartialDependenceDisplay.from_estimator(model, X_val, features=list(order), kind="both")
# kind='both' overlays the per-row ICE curves on the averaged PD curve

Reading a partial dependence curve as causal, or trusting it where the feature's marginal distribution has no data. PD averages over the other features as if this one could be moved independently, so with correlated inputs it evaluates the model at combinations that never occur — and the model's output there is extrapolation, not knowledge. Check the rug or the ICE spread before drawing conclusions from the ends of the curve.