Machine Learning · Module 5 of 26 · 9 concepts · about 16 min

Classification

Predicting a category: turning a linear score into a probability, what a decision boundary is, and why the 0.5 cutoff is a choice you make rather than one the model makes.

At work: Predicting a yes/no or a category: fraud or not, will churn or not, spam or not, which product category.

What makes classification a different problem

What it is

The target is a label from a finite set, so the output space has no ordering or distance. Being wrong is not 'off by an amount' — it is wrong, and different kinds of wrong can cost wildly different amounts.

Why it matters

It changes the loss, the metric and the output. A regression model reports a number; a good classifier reports a probability per class, and the decision — which class to act on — is a separate step taken afterwards, with the costs in hand.

In the real world: Predicting “will this customer churn: yes or no” needs different tools and metrics than predicting how much they'll spend.

Visual example

Regression vs classification

Same pipeline shape, three differences that matter: output, loss, and what 'wrong' costs.

Where the money is

(FP and FN rarely cost the same)

The three real differences

	regression	classification
Output	a real number	a probability per class
Loss	squared / absolute error	log loss (cross-entropy)
Being wrong	off by an amount	a specific confusion, with its own cost

The model gives probabilities; you supply the decision rule

Why this matters

It sets up the whole module: the model's job ends at a probability, and everything about thresholds, precision and recall is the decision layer built on top.

Common misconception

Wrong idea: a classifier outputs a class.

Correct idea: it outputs a score, and predict applies a default cutoff to it. Treating the class as the model's output hides the one knob you most want.

from sklearn.linear_model import LinearRegression, LogisticRegression
import numpy as np

# Regressing on 0/1 labels: predictions escape the [0, 1] range.
LinearRegression().fit(X_train, y_train).predict(X_val[:5])     # e.g. 1.34, -0.21

clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
np.round(clf.predict_proba(X_val[:5])[:, 1], 3)                 # always in [0, 1]

Fitting a linear regression to 0/1 labels and thresholding it. It produces impossible 'probabilities' outside [0, 1], and because squared error penalises confidently-correct points, one distant row can tilt the boundary away from a perfectly good split.

Logistic regression and the sigmoid

What it is

Compute a linear score z = w·x + b, then squash it through the sigmoid 1 / (1 + exp(-z)) to get a number in (0, 1) read as P(y = 1). Fitting minimises log loss, not squared error.

Why it matters

It is the default classifier: fast, calibrated out of the box on reasonable data, and its coefficients are interpretable as log-odds effects. Almost every classification baseline in practice is this model.

In the real world: Banks use it to turn applicant details into a default probability, like 12%, because it is simple and explainable.

Visual example

Sigmoid

The sigmoid maps any real score into (0, 1): steep in the middle, almost flat at the ends.

Reading a coefficient as odds

z = 0 -> p = 0.5, odds 1:1

one more unit of a feature with w = 0.7: z = 0.7 -> p = 0.67

odds multiplied by exp(0.7) = 2.01, at any starting z

Why this matters

Two facts come straight off this curve: the boundary is where the score is zero, and in the saturated tails large changes in the score barely change the probability — which is why confident points contribute almost no gradient.

Common misconception

Wrong idea: the sigmoid is what makes logistic regression non-linear.

Correct idea: the boundary is still linear — a line, plane or hyperplane where z = 0. The sigmoid only reshapes the score into a probability; it does not bend the boundary.

The decision boundary is exactly the set of points where z = 0, because that is where the sigmoid crosses 0.5.

import numpy as np
from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
z = X_val @ clf.coef_[0] + clf.intercept_[0]      # the linear score
p = 1 / (1 + np.exp(-z))                          # the sigmoid
np.allclose(p, clf.predict_proba(X_val)[:, 1])    # True

Reading a coefficient as a change in probability. It is a change in log-odds: +0.7 multiplies the odds by exp(0.7) ≈ 2. The effect on probability depends on where you start — near 0.5 it is large, near 0.99 it is almost nothing.

Log-odds, and what the coefficients mean

What it is

The sigmoid's inverse is the logit: log(p / (1 - p)) = w·x + b. So logistic regression is a linear model of the log-odds, and each coefficient is the additive change in log-odds per unit of its feature.

Why it matters

It makes the model reportable. 'Each extra failed login multiplies the odds of fraud by 1.8' is a sentence a risk team can act on, and it is a direct exponentiation of the coefficient.

In the real world: A coefficient can be read as “each missed payment roughly doubles the odds of default”, a sentence a credit committee understands.

import numpy as np

odds_ratio = np.exp(clf.coef_[0])
for name, ratio in zip(feature_names, odds_ratio):
    print(f"{name}: odds x {ratio:.2f} per unit")

# Interpretable only if the features are scaled the same way they were fitted.

Comparing coefficient magnitudes across unscaled features. A coefficient is per unit of its own feature, so a large one may just mean the feature is measured in small units. Standardise before comparing, and say 'per standard deviation'.

The decision boundary

What it is

The surface in feature space where the model is exactly undecided — for logistic regression, the set of points with w·x + b = 0. On one side the predicted class is 1, on the other 0. In two dimensions it is a line.

Why it matters

It is what a classifier actually is. Accuracy tells you how often the boundary is on the right side of a point; the boundary's shape tells you what the model can and cannot express at all.

In the real world: The line a spam filter draws between spam and not-spam, based on things like links and sender history.

Visual example

Decision boundary

The boundary is a line where the score is zero. Points near it are the uncertain ones, whatever the accuracy says.

class 0class 1boundary: z = 0

Score and probability for three points

z = w0*x0 + w1*x1 + b, with w = (0.9, 0.9), b = 0

(2.4, 1.2) -> z = 3.24 -> p = 0.96: confidently class 1

(-0.6, 1.5) -> z = 0.81 -> p = 0.69: class 1, but barely

Why this matters

It shows where a classifier's errors come from — the shaded strip — and therefore where extra features or a different threshold can actually help.

Common misconception

Wrong idea: a misclassified point means the model is broken.

Correct idea: points near the boundary are supposed to be uncertain. The question is whether the boundary is in the right place, not whether every point is on the right side of it.

Two points sit on the wrong side of the line — the model's errors. Both are close to it, which is where errors mostly live.

import numpy as np

w, b = clf.coef_[0], clf.intercept_[0]
# In 2-D the boundary w0*x0 + w1*x1 + b = 0 rearranges to a line in x0:
x0 = np.linspace(X[:, 0].min(), X[:, 0].max(), 100)
x1 = -(w[0] * x0 + b) / w[1]

# The distance of a point from the boundary is the (scaled) score:
margin = (X_val @ w + b) / np.linalg.norm(w)

Interpreting a boundary drawn on two features when the model was fitted on twenty. The 2-D picture is a slice with the other eighteen features held at some value; the boundary moves as they change, and the slice can look far cleaner than the real problem is.

Log loss (cross-entropy)

What it is

The loss for one example is -log(p) where p is the probability the model gave the *true* class. A confident correct prediction costs almost nothing; a confident wrong one costs unboundedly much.

Why it matters

It is what makes the outputs behave like probabilities. Squared error on labels would reward a timid model that says 0.5 everywhere; log loss punishes miscalibrated confidence specifically, which is why logistic regression's probabilities are usable.

In the real world: Scores how good a fraud model's probabilities are, punishing confident wrong answers the most.

Visual example

Log loss

Confident and wrong is catastrophic. The cost is unbounded as the probability on the true class approaches zero.

Four rows, one metric

true 1, p = 0.9 -> -log(0.9) = 0.105

true 0, p = 0.2 -> -log(0.8) = 0.223

true 1, p = 0.05 -> -log(0.05) = 3.00 <- dominates

mean over the four (with one more at 0.1) = 0.86: worse than 0.693

Why this matters

It explains why log loss and accuracy disagree: accuracy counts sides of the boundary, log loss prices how confident you were, and a model can improve one while wrecking the other.

Common misconception

Wrong idea: log loss is just a smooth stand-in for accuracy.

Correct idea: it measures calibration as well as correctness. Making a correct prediction more confident lowers it; a single overconfident error can raise it more than a dozen cautious ones.

A model that says 0.5 to everything scores 0.693. Any average log loss above that is worse than admitting total ignorance.

import numpy as np
from sklearn.metrics import log_loss

p = clf.predict_proba(X_val)[:, 1]
log_loss(y_val, p)

# By hand, clipping to keep log(0) finite:
pc = np.clip(p, 1e-15, 1 - 1e-15)
float(-np.mean(y_val * np.log(pc) + (1 - y_val) * np.log(1 - pc)))

Predicting a hard 0 or 1 and scoring with log loss. A single confident mistake contributes -log(0) — infinite, or a huge clipped number — and dominates the whole average. Never emit exactly 0 or 1 if log loss is the metric.

predict versus predict_proba

What it is

predict_proba returns one probability per class per row, summing to 1. predict returns a label, which for a binary classifier is predict_proba(...)[:, 1] >= 0.5. decision_function returns the raw score, before any squashing.

Why it matters

The probability is the model's whole output; the label is the probability after a decision rule you did not choose. Keeping the probability lets you rank, set a threshold per cost, abstain on uncertain cases, or combine models.

In the real world: A lending team needs “18% chance of default”, not just “approve”, to set interest rates and limits.

Visual example

Model output

predict is predict_proba plus a cutoff. Move the cutoff and rows 2 and 3 change their answer.

The model's output

(everything else is a rule)

clf.predict(X)

the 0.5 column, nothing more

Five validation rows

	p(class 1)	label at 0.5	label at 0.3
0	0.94	1	1
1	0.62	1	1
2	0.41	0	1
3	0.33	0	1
4	0.08	0	0

(p >= 0.3)

your own rule

Two extra positives, and two new chances to be wrong

Why this matters

It makes the threshold visible as a separate, movable decision — the setup for precision/recall in the next module.

Common misconception

Wrong idea: the model decided these labels.

Correct idea: the model produced the first column. The cutoff produced the labels, and the cutoff was a default nobody chose for your problem.

import numpy as np

p = clf.predict_proba(X_val)[:, 1]
np.array_equal(clf.predict(X_val), (p >= 0.5).astype(int))    # True

np.argsort(-p)[:20]           # the 20 highest-risk rows: a ranking
p[(p > 0.4) & (p < 0.6)]      # cases worth sending to a human

Reading predict_proba column 0 as the positive class. Columns follow clf.classes_, sorted — so for labels {0, 1} the positive class is column 1, and for string labels {"churn", "stay"} it is whichever sorts second. Check classes_ rather than assuming.

Linear separability, and when classes are not separable

What it is

Classes are linearly separable if some hyperplane puts every point of one class on one side. Many real problems are not — concentric rings, XOR-like structure, overlapping populations — and no linear boundary can do better than chance on them.

Why it matters

It decides whether the failure is fixable by tuning or only by changing the hypothesis space. A linear model stuck at 55% on ring-shaped data does not need a better C; it needs features that make the structure linear, or a non-linear model.

In the real world: Fraudsters who mimic normal customers can't be separated by one straight rule; more flexible models are needed.

Visual example

Non-separable

No straight line separates these. The structure is radial, so the fix is a radius feature or a non-linear model.

class 0 (outer ring)class 1 (inner blob)best linear try

One engineered feature fixes it

r2 = x1^2 + x2^2

class 1 (inner): r2 between 0.02 and 0.20

class 0 (outer): r2 about 1.00 -> a single threshold at r2 = 0.5 separates perfectly

Why this matters

It distinguishes 'the model is too simple' from 'the data has no signal' — two diagnoses with completely different remedies, and the second is the expensive mistake.

Common misconception

Wrong idea: a non-linear problem needs a neural network.

Correct idea: it needs a non-linear boundary, which one well-chosen feature can often supply. The radius here turns an impossible problem into a one-dimensional threshold.

The dashed line is the best a linear model can do here: roughly half of each class on each side, so about 50% accuracy.

from sklearn.datasets import make_circles
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
import numpy as np

Xc, yc = make_circles(n_samples=400, noise=0.08, factor=0.45, random_state=0)
LogisticRegression().fit(Xc, yc).score(Xc, yc)          # about 0.5: hopeless
SVC(kernel="rbf").fit(Xc, yc).score(Xc, yc)             # about 0.98

# Or make it linear by hand: radius is the feature that separates rings.
LogisticRegression().fit((Xc ** 2).sum(axis=1).reshape(-1, 1), yc).score(
    (Xc ** 2).sum(axis=1).reshape(-1, 1), yc)

Concluding 'the features carry no signal' from a linear model's poor score. The signal may be perfectly present and simply not linear. Check with a non-linear model before discarding a feature set.

Multi-class strategies: one-vs-rest, one-vs-one, softmax

What it is

One-vs-rest fits one binary classifier per class and takes the highest score. One-vs-one fits one per pair and votes. Multinomial (softmax) logistic regression fits all classes jointly, producing probabilities that sum to 1 across classes.

Why it matters

The strategy changes how many models are fitted, whether the outputs are comparable probabilities, and how ties behave. Softmax gives a coherent distribution; one-vs-rest gives independent scores that need no calibration to be comparable.

In the real world: Routing support tickets into billing, technical, delivery or refund queues is one model choosing among four classes.

Visual example

Multi-class

Three strategies for k classes: k models, k(k-1)/2 models, or one joint model.

Grows quadratically

(20 classes -> 190 models)

k = 5 classes

	models fitted	output	probabilities sum to 1?
One-vs-rest	5	score per class	no
One-vs-one	10	pairwise votes	no
Softmax	1	joint distribution	yes

Softmax when you need probabilities; OvO when each binary fit must be cheap

Model count by k

OvR: k models -> k = 100 gives 100 fits

OvO: k(k-1)/2 -> k = 100 gives 4,950 fits, each on 2 classes only

softmax: 1 fit, k score vectors

Why this matters

The choice is usually driven by cost and by whether downstream code needs a probability distribution — not by accuracy, which is generally close between them.

Common misconception

Wrong idea: multi-class classification needs a special algorithm.

Correct idea: it is mostly a reduction to binary problems, done for you. Which reduction is a real choice with real consequences for the outputs.

from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.svm import LinearSVC

soft = LogisticRegression(max_iter=1000).fit(X_train, y_train)   # multinomial
soft.predict_proba(X_val[:3]).sum(axis=1)                        # ~1.0 per row

OneVsRestClassifier(LinearSVC()).fit(X_train, y_train)     # k models
OneVsOneClassifier(LinearSVC()).fit(X_train, y_train)      # k(k-1)/2 models

Comparing raw one-vs-rest scores as if they were probabilities. Each binary model was fitted against a different 'rest' with a different class balance, so the scores are on incomparable scales — calibrate them, or use softmax.

The 0.5 threshold is a choice, not a law

What it is

predict cuts at 0.5 because it has to cut somewhere. Nothing about 0.5 is optimal: the right cutoff depends on the relative cost of a false positive and a false negative, and on the base rate.

Why it matters

Moving the threshold is the cheapest lever in all of classification. It needs no retraining, no new features and no tuning, and on a skewed problem it changes the useful behaviour of the model more than any algorithm swap will.

In the real world: A hospital screening tool flags patients at 20% risk, not 50%, because missing a sick patient costs far more than an extra check.

Visual example

Thresholding

Every threshold is a different classifier. The count of predicted positives changes by 10x across the range.

Predicted positiveTrue positives caughtFalse positives

Same model, two operating points

t = 0.5: 45 caught of 80 -> recall 0.56, precision 45/74 = 0.61

t = 0.1: 73 caught of 80 -> recall 0.91, precision 73/240 = 0.30

which is better depends entirely on what a missed positive costs

Why this matters

It reframes 'improving the model' — often the gain you want is already available at a different cutoff, and no retraining is involved.

Common misconception

Wrong idea: 0.5 is the neutral, unbiased choice.

Correct idea: 0.5 is only optimal when the two error types cost the same and the model is calibrated. Neither usually holds. Which cutoff to pick is the Model Evaluation module's subject.

There are 80 actual positives in these 1,000 rows. At 0.5 the model finds 45 of them; at 0.1 it finds 73, at the cost of 167 false alarms.

import numpy as np

p = clf.predict_proba(X_val)[:, 1]
for t in [0.2, 0.35, 0.5, 0.65]:
    pred = (p >= t).astype(int)
    tp = int(((pred == 1) & (y_val == 1)).sum())
    fp = int(((pred == 1) & (y_val == 0)).sum())
    fn = int(((pred == 0) & (y_val == 1)).sum())
    print(t, tp, fp, fn)

# Cost-optimal cutoff when a false negative costs 10x a false positive:
# threshold = cost_fp / (cost_fp + cost_fn) = 1 / 11 = 0.09

Tuning the threshold on the test set. It is a parameter like any other: choose it on validation data, using the cost ratio, and then report its performance on test once. Picking the threshold that maximises the test score is the same leak as tuning any hyperparameter there.