Machine Learning · Module 6 of 26 · 10 concepts · about 18 min

Model Evaluation

Choosing and reading the number that decides whether a model ships: what each metric measures, what it hides, and how to compute it from a confusion matrix by hand.

At work: Deciding whether a model is good enough to use, in terms the business cares about: fraud caught, customers wrongly blocked, money saved.

The confusion matrix

What it is

A table of counts with actual classes down the rows and predicted classes across the columns. For a binary problem it has four cells: true negatives, false positives, false negatives and true positives. Every classification metric is a ratio of these four numbers.

Why it matters

It is the only classification report that loses no information. Accuracy, precision, recall, F1 and specificity are all summaries of this table, so reading the table first tells you which summary is worth quoting.

In the real world: A fraud model's results laid out: frauds caught, frauds missed, good customers wrongly blocked, and good customers correctly let through.

Visual example

Confusion matrix

Four counts. The diagonal is correct, the off-diagonal is the two kinds of mistake — and they rarely cost the same.

False negatives

(35 missed positives)

False positives

(29 false alarms)

1,000 validation rows, threshold 0.5

	pred 0	pred 1
actual 0	891	29
actual 1	35	45

precision = 45/(45+29)

of those flagged, how many were right

recall = 45/(45+35)

of the real positives, how many were found

cm.ravel() → 891, 29, 35, 45

Every metric from these four numbers

accuracy = (891 + 45) / 1000 = 0.936

precision = 45 / 74 = 0.608

recall = 45 / 80 = 0.563

specificity = 891 / 920 = 0.968

Why this matters

Quoting one metric hides three. The table is small enough to read in full, and reading it in full is what prevents the accuracy paradox further down this module.

Common misconception

Wrong idea: the confusion matrix is a debugging aid; the metric is the result.

Correct idea: the matrix is the result. Every metric is a lossy compression of it, chosen to match a decision — so pick the compression after seeing the table, not before.

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

cm = confusion_matrix(y_val, clf.predict(X_val))
tn, fp, fn, tp = cm.ravel()      # only valid for the binary case
tn, fp, fn, tp

confusion_matrix(y_val, clf.predict(X_val), normalize="true")   # row-wise rates

Mixing up the order. scikit-learn's rows are the actual class and columns the prediction — the opposite of many textbooks — and ravel() gives tn, fp, fn, tp, not tp first. Reversing it silently swaps precision and recall.

Practise it: Macro and Micro F1, From the Confusion Matrix Alone, Precision, Recall and F1 by Hand

Accuracy, precision, recall and F1

What it is

Accuracy is the share of all predictions that were right. Precision is TP / (TP + FP) — how trustworthy a positive prediction is. Recall is TP / (TP + FN) — how much of the positive class you found. F1 is their harmonic mean.

Why it matters

Precision and recall answer different questions and trade off against each other, so naming which one the decision needs is the core evaluation skill. Precision matters when acting on a positive is expensive; recall matters when missing one is.

In the real world: Precision is how many flagged transactions were really fraud; recall is how much of all fraud was caught. Banks watch both.

Visual example

Metric arithmetic

The same four counts, compressed four ways. Accuracy is 0.936 and recall is 0.563 — both about the same model.

Flattered by 891 TNs

(the easy majority)

TN 891 FP 29 FN 35 TP 45

	what it asks	value
Accuracy	of all rows, how many right?	0.936
Precision	of rows I flagged, how many right?	0.608
Recall	of real positives, how many found?	0.563
F1	harmonic mean of the two above	0.584

Accuracy is the only one of the four that counts TN

Worked, digit by digit

accuracy = (TN + TP)/N = (891 + 45)/1000 = 0.936

precision = TP/(TP + FP) = 45/(45 + 29) = 0.608

recall = TP/(TP + FN) = 45/(45 + 35) = 0.563

F1 = 2PR/(P + R) = 2(0.608)(0.563)/1.171 = 0.584

Why this matters

The gap between 0.936 and 0.563 is the whole reason to know the definitions: the first number is what a dashboard reports, the second is what the positive class experiences.

Common misconception

Wrong idea: high accuracy means the model is good at the task.

Correct idea: accuracy includes the true negatives, which on a skewed problem are nearly free. Precision and recall ignore them — which is precisely why they are the informative pair.

from sklearn.metrics import (accuracy_score, precision_score,
                             recall_score, f1_score, classification_report)

pred = clf.predict(X_val)
accuracy_score(y_val, pred), precision_score(y_val, pred), recall_score(y_val, pred)
f1_score(y_val, pred)
print(classification_report(y_val, pred, digits=3))

Quoting F1 because it 'balances' the two. The harmonic mean weights precision and recall equally, which is a strong and usually unexamined claim about your costs. If a missed fraud costs 50 times a false alarm, F1 is the wrong summary — use fbeta_score or the cost directly.

Practise it: Macro and Micro F1, From the Confusion Matrix Alone, Precision, Recall and F1 by Hand, How Wrong Are Your Probabilities?, Pick a Threshold for an Imbalanced Classifier

The precision-recall tradeoff and the PR curve

What it is

Lowering the threshold flags more rows: recall rises and precision generally falls. Plotting precision against recall over every threshold gives the PR curve, summarised by average precision (the area under it).

Why it matters

On imbalanced problems the PR curve is the honest summary, because both axes depend only on the positive class. It is also the plot you pick an operating point from, by finding the recall you need and reading off the precision you will get.

In the real world: Catching more fraud means blocking more good customers; the curve shows the trade so the business can pick its point.

Visual example

PR curve

Precision falls as recall rises. The curve is the menu of operating points one model offers.

Model (AP = 0.52)Random (= base rate 0.08)

Reading an operating point off the curve

need recall 0.80 -> precision 0.33 -> 2 false alarms per true hit

at recall 0.56 -> precision 0.61 -> roughly 3 hits per 2 alarms

with 80 positives, recall 0.80 means 64 found and about 130 flagged

Why this matters

It converts 'is the model good?' into 'at the recall we need, is the precision affordable?' — a question the business can answer and the metric cannot.

Common misconception

Wrong idea: the model has one precision and one recall.

Correct idea: it has a curve. Precision and recall are properties of the threshold, not of the model, and only the curve is a property of the model.

The dashed line is what random guessing achieves: precision equal to the positive rate, 0.08, at every recall.

from sklearn.metrics import precision_recall_curve, average_precision_score
import numpy as np

p = clf.predict_proba(X_val)[:, 1]
prec, rec, thresh = precision_recall_curve(y_val, p)
average_precision_score(y_val, p)

# The threshold that reaches at least 0.80 recall, with the best precision:
ok = rec[:-1] >= 0.80
float(thresh[ok][np.argmax(prec[:-1][ok])])

Comparing PR curves across datasets. The baseline of a PR curve is the positive rate, so average precision of 0.4 is excellent at a 2% positive rate and poor at a 40% one. Always quote the base rate beside it.

Practise it: Macro and Micro F1, From the Confusion Matrix Alone, Precision, Recall and F1 by Hand, Pick a Threshold for an Imbalanced Classifier

The ROC curve and AUC

What it is

The ROC curve plots the true positive rate against the false positive rate across all thresholds. AUC is the area beneath it, and equals the probability that a random positive is ranked above a random negative.

Why it matters

AUC is threshold-free and scale-free, so it measures ranking quality alone — useful for comparing models before an operating point exists, and for a metric that does not move when the class balance changes.

In the real world: A single number to compare two credit models' ability to rank risky applicants above safe ones.

Visual example

ROC curve

AUC measures ranking. The dashed diagonal is random guessing; the distance above it is the signal.

Good model (AUC 0.88)Weak model (AUC 0.66)Random (AUC 0.50)

AUC as a ranking probability

AUC 0.88 = 88% of (positive, negative) pairs are ranked correctly

at FPR 0.10 with 920 negatives: 92 false positives

TPR 0.62 with 80 positives: 50 found -> precision 50/142 = 0.35

Why this matters

It shows what AUC does and does not tell you: the curve's shape near the left edge is the region any real deployment lives in, and the single area number averages over regions you will never use.

Common misconception

Wrong idea: AUC 0.9 means the model is right 90% of the time.

Correct idea: it means a random positive outranks a random negative 90% of the time. It says nothing about accuracy at any threshold, and nothing about precision.

A curve below the diagonal is not a bad model — it is a model with its sign flipped. Invert the scores and AUC becomes 1 - AUC.

from sklearn.metrics import roc_curve, roc_auc_score

p = clf.predict_proba(X_val)[:, 1]
fpr, tpr, thresh = roc_curve(y_val, p)
roc_auc_score(y_val, p)      # 0.5 = random ranking, 1.0 = perfect

# Interpretation check: AUC is a ranking statistic, not an accuracy.

Trusting AUC on a heavily imbalanced problem. The false positive rate has a huge denominator, so a thousand extra false positives among a million negatives barely move it — the curve looks excellent while precision at any usable threshold is dreadful. Use the PR curve there.

Choosing the operating point

What it is

Plot precision and recall against the threshold, decide which constraint is binding — a minimum precision the business will tolerate, or a minimum recall compliance requires — and pick the threshold that satisfies it on validation data.

Why it matters

It turns evaluation into a decision. Without this step a model has no defined behaviour; with it, the deployed system has a stated guarantee, and you know which of the two error types you have chosen to absorb.

In the real world: A card issuer picks the fraud threshold where investigators have enough capacity to review every flagged case.

Visual example

Operating point

Precision and recall cross. Where you sit on this plot is the deployment decision, not a model property.

PrecisionRecallF1

Expected cost at three thresholds (FP = 1, FN = 12)

t = 0.5: 29 FP + 35 FN -> 29 + 420 = 449

t = 0.2: 96 FP + 16 FN -> 96 + 192 = 288 <- cheapest

t = 0.1: 167 FP + 7 FN -> 167 + 84 = 251 ... recompute as costs change

Why this matters

It is the only place the cost of an error enters the pipeline explicitly. Every model above this line of the plot is the same model; the difference is what you have chosen to be wrong about.

Common misconception

Wrong idea: the best threshold is the one with the best metric.

Correct idea: the best threshold is the one with the lowest cost under your error prices. Maximising F1 is what you do when you cannot state those prices.

F1 peaks at 0.30, but a policy requiring 80% recall forces 0.20 — and costs 9 points of precision to get there.

import numpy as np
from sklearn.metrics import precision_recall_curve

prec, rec, thresh = precision_recall_curve(y_val, p)

# Cheapest threshold under an explicit cost ratio.
cost_fp, cost_fn = 1.0, 12.0
costs = [(cost_fp * ((p >= t) & (y_val == 0)).sum()
          + cost_fn * ((p < t) & (y_val == 1)).sum()) for t in thresh]
float(thresh[int(np.argmin(costs))])

Choosing the threshold that maximises F1 by default. That implicitly declares a false positive and a false negative equally costly. If you can state the cost ratio at all, minimise expected cost instead — it usually lands somewhere quite different.

The accuracy paradox on imbalanced data

What it is

When one class is rare, a model that always predicts the majority class achieves high accuracy while providing no value at all. The rarer the positives, the higher that useless score.

Why it matters

It is the most common way a genuinely bad model passes review. Every imbalanced project needs the majority baseline computed and shown next to the model's accuracy, or the number means nothing.

In the real world: A model that says “never fraud” is 99.9% accurate and catches nothing, which is why accuracy alone misleads.

Visual example

Accuracy paradox

99.2% accuracy, zero positives found. The rarer the class, the more flattering accuracy becomes.

Worse on accuracy

(and vastly more useful)

10,000 rows, 80 positives (0.8%)

	accuracy	recall	positives caught
Always predict 0	0.992	0	0
Real model	0.988	0.56	45
Difference	-0.004	+0.56	+45

Accuracy ranks these two models backwards

Why the useful model scores lower

always-0: 9,920 correct of 10,000 = 0.992, and 0 of 80 positives

real model: 45 TP + 9,835 TN = 9,880 -> 0.988, with 85 false alarms

the 85 false alarms cost 0.0085 of accuracy and bought 45 catches

Why this matters

It is the concrete case for never quoting bare accuracy on skewed data — the metric does not merely understate the model, it prefers the useless one.

Common misconception

Wrong idea: accuracy is a neutral starting point that you refine later.

Correct idea: on skewed data it is actively misleading: it can rank a valuable model below a constant predictor, which no amount of later refinement undoes.

from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, recall_score, average_precision_score

dumb = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
accuracy_score(y_val, dumb.predict(X_val))      # 0.992 at a 0.8% positive rate
recall_score(y_val, dumb.predict(X_val))        # 0.0 — it never says yes
average_precision_score(y_val, p)               # the metric that notices

Reporting a 0.4-point accuracy improvement over the majority baseline as a success. At a 0.8% positive rate, moving accuracy from 0.992 to 0.996 could mean catching half the positives — or it could mean nothing. Only the confusion matrix distinguishes them.

Practise it: Pick a Threshold for an Imbalanced Classifier

Regression metrics: MAE, MSE, RMSE and R²

What it is

MAE is the mean absolute residual, in the units of y. MSE is the mean squared residual; RMSE is its square root, also in the units of y but outlier-sensitive. R² is the share of variance explained relative to predicting the mean.

Why it matters

They answer different questions. RMSE for 'how wrong am I, in pounds, with big misses counted extra'; MAE for typical error; R² for 'am I beating the mean at all', which is the only one that is comparable across datasets — and only loosely.

In the real world: “Our delivery estimates are off by 6 minutes on average” (MAE) is a number an operations manager can act on.

Visual example

Regression metrics

Predicted against actual, with y = x as perfect. The vertical gap to that line is the residual each metric sums.

Validation rowsPerfect prediction

The same twelve residuals, four ways

MAE = mean|resid| = (4+3+4+4+5+5+3+5+4+4+6+30)/12 = 6.4

MSE = mean(resid^2) = 89.8 -> RMSE = 9.5

RMSE/MAE = 1.48: well above 1, so the errors are skewed by outliers

R^2 = 1 - 89.8/var(y) = 1 - 89.8/690 = 0.87

Why this matters

The RMSE-to-MAE ratio is a free diagnostic: near 1 means uniform errors, much larger means a few rows own the loss — and those rows are worth looking at individually.

Common misconception

Wrong idea: RMSE and MAE measure the same thing on different scales.

Correct idea: RMSE is outlier-weighted and MAE is not. When they diverge, the divergence itself is the finding: your error distribution has a tail.

Eleven rows sit within 4 of the diagonal; one sits 30 below it. MAE barely notices that row; RMSE is dominated by it.

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

pred = model.predict(X_val)
mean_absolute_error(y_val, pred)
rmse = np.sqrt(mean_squared_error(y_val, pred))
r2_score(y_val, pred)         # 0 = as good as the mean; negative = worse

Believing R² cannot be negative. On held-out data it very much can: a model that does worse than predicting the training mean gives a negative R², and that is the clearest possible signal of overfitting.

Macro, micro and weighted averaging

What it is

For multi-class metrics, macro averages the per-class scores equally, micro pools all the counts before computing (equal to accuracy for single-label problems), and weighted averages the per-class scores by class support.

Why it matters

The choice decides whether rare classes count. Macro gives a 10-row class the same weight as a 10,000-row one, which is what you want when every class matters; micro and weighted let the big classes dominate.

In the real world: A ticket-routing model can look great overall while failing the rare “legal” category; macro averaging exposes it.

Visual example

Multi-class averaging

Same three per-class F1 scores, three averages 0.25 apart. The rare class either counts or it does not.

Basically broken

(and nearly invisible)

Per-class F1 and support

	support	F1
class A	900	0.94
class B	80	0.71
class C (rare)	20	0.12

macro 0.59 · weighted 0.91 · micro 0.92

The three averages, computed

macro = (0.94 + 0.71 + 0.12)/3 = 0.59

weighted = (900(0.94) + 80(0.71) + 20(0.12))/1000 = 0.91

micro pools TP/FP/FN over all classes -> 0.92, i.e. accuracy

Why this matters

Reporting 0.91 and reporting 0.59 for the same model are both defensible and lead to opposite decisions. Saying which average you used is not pedantry.

Common misconception

Wrong idea: the averaging mode is a formatting detail.

Correct idea: it is a statement about what matters. Macro says every class counts; weighted says every row counts. They differ most exactly where you care most.

from sklearn.metrics import f1_score, classification_report

f1_score(y_val, pred, average="macro")      # every class counts equally
f1_score(y_val, pred, average="micro")     # every row counts equally
f1_score(y_val, pred, average="weighted")  # per class, weighted by support
print(classification_report(y_val, pred))   # the per-class numbers behind them

Reporting a weighted F1 on a skewed multi-class problem and describing the model as balanced. Weighted averaging hides a completely failed rare class behind the dominant one — exactly the class you probably built the model for.

Practise it: Macro and Micro F1, From the Confusion Matrix Alone

Calibration

What it is

A model is calibrated when its stated probabilities match observed frequencies: among rows given 0.30, about 30% are actually positive. The reliability curve plots predicted probability against observed rate, bin by bin.

Why it matters

Any decision that uses the probability as a number — expected cost, expected revenue, a risk-weighted queue — is wrong if the model is miscalibrated, even when its ranking (and therefore its AUC) is perfect.

In the real world: When an insurance model says 10% risk, about 10 in 100 such customers should really claim, or premiums are wrong.

Visual example

Calibration

Overconfident at both ends. Rows the model calls 0.9 are positive only 72% of the time.

Uncalibrated modelAfter isotonic calibrationPerfectly calibrated

What miscalibration costs a decision

1,000 rows scored 0.95; expected positives = 950

actual positives = 720 -> the plan over-books by 230

after calibration the same bin predicts 0.93 and observes 0.93

Why this matters

It separates two qualities people bundle together: ranking (AUC) and probability accuracy (Brier, calibration). A model can be perfect at one and useless at the other.

Common misconception

Wrong idea: a high-AUC model gives trustworthy probabilities.

Correct idea: AUC only checks the order. Calibration checks the values, and the values are what any expected-cost calculation multiplies by.

Both models rank identically — calibration is monotonic, so AUC is unchanged. Only the numbers on the scores move.

from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss

observed, predicted = calibration_curve(y_val, p, n_bins=10, strategy="quantile")
brier_score_loss(y_val, p)

cal = CalibratedClassifierCV(clf, method="isotonic", cv=5).fit(X_train, y_train)

Calibrating on the data the model was fitted on. The training probabilities are already optimistic, so the calibrator learns the wrong correction. CalibratedClassifierCV does it out-of-fold for you; hand-rolled calibration usually does not.

Practise it: How Wrong Are Your Probabilities?

Cross-validation as the evaluation protocol

What it is

Split the training data into k folds; fit on k-1 and score on the held-out one, k times. Report the mean and the spread. StratifiedKFold preserves class balance, GroupKFold keeps an entity in one fold, TimeSeriesSplit respects order.

Why it matters

A single validation split gives one noisy number, and on small data the noise is larger than the differences you are trying to detect. Cross-validation gives a mean and a standard deviation, which is what makes 'better' a checkable claim.

In the real world: Testing a churn model on five different slices of customers gives a steadier answer than one lucky split.

Visual example

Cross-validation

Five folds, five scores. The spread is what tells you whether a 0.01 improvement is real.

The column that decides

(gain vs spread)

5-fold average precision, two models

	mean	std	verdict
Model A	0.512	0.031	—
Model B	0.524	0.029	+0.012
Difference			inside the noise

A gain of 0.012 against a std of 0.03 is not yet a result

Fold-by-fold, model B minus model A

folds: +0.05, -0.02, +0.03, -0.01, +0.01

mean +0.012, and the sign flips on 2 of 5 folds

to detect 0.012 reliably you need more data or more folds, not more tuning

Why this matters

It is the discipline that stops a project from chasing noise: the reported difference has to exceed the fold-to-fold variation before it means anything.

Common misconception

Wrong idea: cross-validation is for small datasets; with lots of data a single split is fine.

Correct idea: a single split is fine when the split noise is smaller than the effects you care about. That is a comparison you can only make once you have measured the noise.

from sklearn.model_selection import cross_validate, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
res = cross_validate(pipe, X_train, y_train, cv=cv,
                     scoring=["average_precision", "roc_auc"], return_train_score=True)
res["test_average_precision"].mean(), res["test_average_precision"].std()

Cross-validating a bare estimator on pre-transformed data. Each fold's validation rows contributed to the scaler or encoder, so all k scores are optimistic — pass a Pipeline to cross_validate so preprocessing refits inside every fold.

Practise it: A Pipeline That Does Not Leak, A Split That Does Not Leak Across Groups