Machine Learning · Module 13 of 26 · 7 concepts · about 13 min

SVM & Naive Bayes

Two more classifiers after kNN: one that draws the boundary with the widest possible margin, and one that simply counts evidence.

At work: Two more ways to predict, each still useful: clear-margin classifiers, and fast text filters like spam detection.

Support Vector Machines: the maximum margin

What it is

Among all the hyperplanes that separate two classes, an SVM picks the one whose distance to the nearest point of either class is largest. That distance is the margin; the points sitting exactly on the margin's edges are the support vectors, and they alone determine the boundary.

Why it matters

Maximising the margin is a principled answer to 'which of the infinitely many separating lines should I use', and it generalises well: a wide margin means the boundary can move a long way before any training point changes side. It is also why the model is so compact — everything but the support vectors can be discarded.

In the real world: Drawing the widest possible gap between genuine and fake product reviews, which holds up well on new reviews.

Visual example

SVC(kernel='linear')

Two classes, the maximum-margin boundary, and the shaded margin. Only the three circled support vectors touch it.

Class 0Class 1Decision boundaryMargin edgesMargin edge (class 1 side)

The margin, numerically

boundary at x1 = 0.05; nearest class-0 point at x1 = -0.6, nearest class-1 at x1 = 0.7

margin half-width = 0.65, so the full margin is 1.30 wide

any other separating line would have one class closer than 0.65

3 support vectors out of 16 points: the other 13 could be deleted

Why this matters

Almost every training point is irrelevant to the fitted boundary. That is what 'support vector' means, and it is why an SVM is robust to points far from the boundary and extremely sensitive to points on it.

Common misconception

Wrong idea: the SVM boundary is the line that best separates the bulk of the two clouds.

Correct idea: it is determined entirely by the few closest points. Add a thousand class-0 rows far to the left and the boundary does not move a millimetre.

1Find candidatesEvery hyperplane that separates the classes (with slack, if none does).
2Maximise the gapPick the one whose nearest point on either side is farthest away.
3Keep the supportOnly the points on the margin edges enter the final model.

from sklearn.svm import SVC

svc = make_pipeline(StandardScaler(), SVC(kernel="linear", C=1.0)).fit(X_train, y_train)
model = svc[-1]
model.support_vectors_.shape      # (34, 30) — only 34 of 455 rows matter
model.n_support_                  # per class
model.decision_function(svc[0].transform(X_test[:5])).round(3)
# signed distance to the hyperplane; the sign is the prediction

Calling SVC(probability=True) and treating the output as a real probability. It fits a separate Platt-scaling logistic model by internal 5-fold CV, which is slow, can disagree with predict on borderline points, and is only as good as that calibration. If you need ranking, use decision_function; if you need calibrated probabilities, use CalibratedClassifierCV deliberately.

Soft margins and C

What it is

Real data is rarely separable, so the SVM allows points to violate the margin at a per-point penalty. C is the price of a violation: large C means violations are expensive, giving a narrow margin that bends to fit every point; small C means they are cheap, giving a wide margin that tolerates errors.

Why it matters

C is the SVM's regularisation strength, inverted — it is 1/lambda, not lambda. Getting the direction wrong is the single commonest SVM mistake, because the number that makes the model *simpler* is the small one.

In the real world: Allowing a few mistakes so the classifier isn't thrown off by mislabelled training emails.

for C in [0.01, 0.1, 1, 10, 100]:
    svc = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=C, gamma="scale"))
    s = cross_val_score(svc, X, y, cv=5, scoring="roc_auc")
    print(C, round(s.mean(), 4), svc.fit(X, y)[-1].n_support_.sum())
# 0.01 -> 0.9712, 268 SVs (wide margin: most points violate it)
# 1    -> 0.9948,  78 SVs
# 100  -> 0.9903,  41 SVs (narrow margin, contorted boundary)

Searching C on a linear grid like [1, 2, 3, ..., 10]. Its effect is multiplicative, so the meaningful grid is logarithmic — np.logspace(-3, 3, 7) — and it must be searched *jointly* with gamma, because the two interact strongly: a large gamma with a large C overfits far more than either alone.

The kernel trick

What it is

The SVM's solution depends on the data only through inner products between pairs of points. Replace that inner product with a kernel function K(x, x') and you are fitting a linear boundary in a higher-dimensional feature space — without ever computing coordinates in it. The RBF kernel exp(-gamma * ||x - x'||^2) corresponds to an infinite-dimensional space.

Why it matters

It buys nonlinearity at the cost of one function substitution. A boundary that is a circle in the original two features is a plane in the space of (x1, x2, x1² + x2²), and the kernel lets you work in that space at the price of the original one.

In the real world: Separating complex patterns, like fraud that isn't a straight-line rule, without building thousands of features by hand.

Visual example

RBF / polynomial kernel

A circular boundary in (x1, x2) becomes a flat threshold in the single feature r² = x1² + x2². The kernel does this implicitly.

Class 1 (inside)Class 0 (outside)Linear boundary in the new space

The same point, two spaces

point (1.5, 1.0): in (x1, x2) it sits among class-0 points at similar x1

r² = 1.5² + 1.0² = 3.25, comfortably below the threshold 4.9

point (1.3, 2.1): r² = 1.69 + 4.41 = 6.10, above it -> class 0

one linear inequality in r² replaces a circle in (x1, x2)

Why this matters

The two classes overlap completely along the x-axis and separate perfectly along the y-axis, which is the engineered feature. That vertical separation is exactly what a kernel manufactures — and what the next module builds by hand.

Common misconception

Wrong idea: the kernel trick projects your data into a higher-dimensional space.

Correct idea: it computes the inner products that space would have, without ever building the coordinates. An RBF kernel's space is infinite-dimensional, so building it is not an option.

rng = np.random.default_rng(0)
X = rng.uniform(-3, 3, size=(600, 2))
y = (X[:, 0]**2 + X[:, 1]**2 < 4).astype(int)   # a circular boundary

SVC(kernel="linear").fit(X, y).score(X, y)      # ~0.66 — no line can do this
SVC(kernel="rbf", gamma="scale").fit(X, y).score(X, y)   # ~0.99

# the same thing by hand, with one engineered feature:
X3 = np.c_[X, (X**2).sum(axis=1)]
SVC(kernel="linear").fit(X3, y).score(X3, y)    # ~0.99 — a PLANE at r^2 = 4

Reaching for kernel='rbf' by default on wide data. With many features the data is nearly separable by a hyperplane already, and an RBF kernel there mostly adds variance and cost — LinearSVC (or SGDClassifier with a hinge loss) is both faster and usually better. Kernel SVMs are also O(n²)–O(n³) in the number of rows, so above roughly 100k rows they become impractical regardless.

gamma: the RBF kernel's reach

What it is

gamma in exp(-gamma * ||x - x'||²) sets how quickly influence decays with distance. Small gamma gives each support vector a wide, smooth zone of influence; large gamma makes it a narrow bump, so the boundary becomes a collection of islands around individual training points.

Why it matters

gamma and C are the SVM's two-dimensional capacity knob and they must be tuned together. Large gamma alone overfits regardless of C; the default gamma='scale' (1 / (n_features * X.var())) is a good, scale-aware starting point that gamma='auto' and hand-set values often are not.

In the real world: Setting how locally the model reacts; too local and it memorises individual customers.

Visual example

exp(-gamma * d²)

The RBF kernel's value against distance, at three gamma values. **gamma is the width of a support vector's influence.**

gamma = 0.05 (smooth, underfits)gamma = 0.5 (the working range)gamma = 5 (islands, overfits)

Influence radius = 1 / sqrt(gamma)

gamma = 5 -> radius 0.45: a support vector affects only its immediate neighbours

gamma = 0.5 -> radius 1.41: overlapping zones, a smooth curved boundary

gamma = 0.05 -> radius 4.47: wider than the data, so the boundary is near-linear

standardised features have typical pairwise distance ~sqrt(2 * n_features)

Why this matters

Comparing the radius to the typical distance between your points tells you immediately whether a gamma is sane: much smaller and the model memorises, much larger and it is linear.

Common misconception

Wrong idea: gamma controls how curved the boundary is.

Correct idea: it controls how far each support vector's vote carries. The curvature is a consequence: narrow votes make a bumpy boundary, wide votes make a flat one.

from sklearn.model_selection import GridSearchCV

search = GridSearchCV(
    make_pipeline(StandardScaler(), SVC(kernel="rbf")),
    {"svc__C": np.logspace(-2, 3, 6), "svc__gamma": np.logspace(-4, 1, 6)},
    cv=5, scoring="roc_auc", n_jobs=-1,
).fit(X, y)
search.best_params_        # e.g. {'svc__C': 10.0, 'svc__gamma': 0.01}

Tuning gamma without scaling first. gamma multiplies a squared distance, and an unscaled column makes those distances enormous, so every sensible gamma collapses the kernel to zero and every prediction becomes the majority class. Scale, then tune.

Naive Bayes and the independence assumption

What it is

Apply Bayes' rule and assume the features are conditionally independent given the class: P(y | x) ∝ P(y) * prod_j P(x_j | y). That assumption reduces fitting to counting one small distribution per feature per class, so training is a single pass over the data.

Why it matters

The assumption is nearly always false, and the model works well anyway. For *classification* you only need the correct argmax, not correct probabilities — dependent features multiply their evidence in twice, which distorts the magnitudes badly while often leaving the ranking of classes intact.

In the real world: The fast, simple method behind many early spam filters, still used as a quick baseline for text.

Visual example

GaussianNB

One row scored by GaussianNB. Everything is a product of per-feature likelihoods, and two correlated features each get a full vote.

corr = 0.97

(the same evidence, counted twice)

log P(x_j | y) for one test row

feature	class 0	class 1	contribution
log P(y)	-0.98	-0.47	-0.51 to class 1
worst radius	-4.12	-1.88	-2.24 to class 1
mean radius	-4.05	-1.91	-2.14 to class 1
texture	-2.31	-2.44	+0.13 to class 0
— total —	-11.46	-6.7	class 1

predict_proba

1 / (1 + e^(-4.76)) = 0.9915

Independence makes the log-likelihood a sum, which is why it is so fast

Double-counting, quantified

the two radius columns are 0.97 correlated: essentially one measurement

each contributes ~2.2 of log-odds, so the evidence is counted twice

honest single count -> log-odds 2.6 -> P = 0.93

naive count -> log-odds 4.8 -> P = 0.99. Same class, very different confidence.

Why this matters

The prediction survives the violated assumption; the probability does not. That asymmetry is the whole reason Naive Bayes remains useful in 2026 despite an assumption almost no dataset satisfies.

Common misconception

Wrong idea: Naive Bayes is unusable because its independence assumption is false.

Correct idea: the assumption breaks the magnitudes, not usually the argmax. It stays a strong, essentially untunable, single-pass baseline.

from sklearn.naive_bayes import GaussianNB, MultinomialNB

nb = GaussianNB().fit(X_train, y_train)
nb.theta_.shape           # (n_classes, n_features) — one mean per feature per class
nb.var_.shape             # and one variance
nb.class_prior_           # P(y), from the training frequencies

nb.predict_proba(X_test[:2]).round(4)   # often 0.9999 / 0.0001: overconfident

Using its probabilities for anything quantitative — a threshold, an expected value, a cost calculation. Correlated features make Naive Bayes wildly overconfident, so predict_proba saturates at 0 and 1 and is poorly calibrated. It is a fine classifier and a bad probability model; calibrate it if you need the numbers.

Gaussian, Multinomial and Bernoulli variants

What it is

The variants differ only in what P(x_j | y) is assumed to be. GaussianNB fits a normal per feature per class, for continuous features. MultinomialNB models counts, for term frequencies or TF-IDF. BernoulliNB models binary presence/absence and, unlike Multinomial, explicitly scores the features that are *absent*. CategoricalNB handles nominal categories directly.

Why it matters

Choosing the wrong variant is the most common way Naive Bayes goes wrong. MultinomialNB on standardised (and therefore negative) features raises; GaussianNB on a sparse bag-of-words matrix is both badly mis-specified and forces the matrix dense.

In the real world: Word counts in support tickets suit one variant; yes/no flags on a form suit another.

Visual example

Choosing a variant

Which variant for which data, and the raise-or-mislead you get from the wrong one.

variant -> assumed P(x_j | y)

variant	feature type	P(x_j \| y)	wrong-use symptom
GaussianNB	continuous	Normal(mu, var)	poor fit on skewed columns
MultinomialNB	counts, TF-IDF	relative frequency	raises on negative values
BernoulliNB	binary 0/1	Bernoulli(p)	ignores counts above 1
CategoricalNB	nominal codes	per-category table	needs 0..k-1 encoding

StandardScaler() + MultinomialNB()

ValueError: Negative values in data

The variant *is* the model; there is little else to tune but alpha

Bernoulli vs Multinomial on the same document

doc = 'good good good film', vocabulary {good, bad, film}

Multinomial sees counts [3, 0, 1]: 'good' contributes three times

Bernoulli sees presence [1, 0, 1]: 'good' contributes once

Bernoulli also scores 'bad' as ABSENT, which is evidence Multinomial discards

Why this matters

Naive Bayes has almost no capacity knobs, so nearly all of its accuracy comes from picking the right likelihood for the data you have.

Common misconception

Wrong idea: the variants are interchangeable implementations of the same model.

Correct idea: each assumes a different distribution per feature. Bernoulli and Multinomial disagree about whether a repeated word is stronger evidence, and both are right for different data.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB, BernoulliNB

text_clf = make_pipeline(TfidfVectorizer(min_df=2, ngram_range=(1, 2)),
                         MultinomialNB(alpha=0.1))
cross_val_score(text_clf, docs, labels, cv=5, scoring="f1_macro").mean()

# short documents, where presence matters more than frequency:
make_pipeline(TfidfVectorizer(binary=True), BernoulliNB(alpha=1.0))

Leaving alpha=0. Laplace smoothing is what stops an unseen (feature, class) pair from producing P = 0, and one zero in a product makes the whole posterior zero — a single unseen word can then veto a class outright. Keep alpha > 0 and tune it; it is the only real hyperparameter the model has.

Choosing between the three

What it is

Naive Bayes is the fastest and most biased: one pass, no tuning, strong on wide sparse text. KNN needs no assumptions but needs scaling, low dimensionality and memory at prediction time. A kernel SVM is the most accurate of the three on small-to-medium dense numeric data and the most expensive to fit and tune.

Why it matters

All three are cheap enough to run as a baseline panel before a forest or a GBM, and the pattern of which ones do well is itself diagnostic: if KNN does well your feature geometry is sound, if Naive Bayes does well the features are close to independent evidence, if only the RBF SVM does well the boundary is smooth and nonlinear.

In the real world: Picking KNN for recommendations, SVM for clean medium-sized data, and Naive Bayes for fast text triage.

models = {
    "nb": GaussianNB(),
    "knn": make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=9)),
    "svm": make_pipeline(StandardScaler(), SVC(kernel="rbf", C=10, gamma=0.01)),
}
for name, m in models.items():
    s = cross_val_score(m, X, y, cv=cv, scoring="roc_auc", n_jobs=-1)
    print(f"{name:4s} {s.mean():.4f} +/- {s.std():.4f}")

Dismissing all three because a gradient-boosted model beat them. On wide sparse text a linear SVM or MultinomialNB frequently beats a GBM outright and trains in seconds; and in a latency-bound service, a Naive Bayes model that is 1 point worse and 200x faster is often the correct engineering answer.