Machine Learning · Module 14 of 26 · 10 concepts · about 19 min

Feature Engineering

Building the columns the model cannot build for itself — and the out-of-fold discipline that keeps the good ones from being leaks.

At work: Turning raw business data into signals a model can use: days since last order, spend per visit, time of day. Often the biggest win.

What feature engineering is, and why it still matters

What it is

Constructing new input columns from existing ones — and from domain knowledge that is nowhere in the table — so that the relationship the model has to learn becomes one it can represent easily. It is a change to the hypothesis space, not to the algorithm.

Why it matters

Every model has a representational limit. A linear model cannot multiply two features; a tree needs exponentially many axis-aligned splits to imitate one sum; no model at all can know that two timestamps are 23 hours apart on a 24-hour clock. A feature that encodes such a fact hands the model something no amount of capacity would find.

In the real world: Adding “days since last order” often improves a churn model more than switching to a fancier algorithm.

Visual example

Before / after

The same three raw columns, plus three derived ones. The model's capacity did not change; the hypothesis space did.

cross-validated R², cumulative

feature set	n cols	CV R²	gain
raw 3 columns	3	0.612	-
+ kg_per_km (a ratio)	4	0.688	+0.076
+ over_legal_hours (a threshold)	5	0.719	+0.031
+ is_weekend (a calendar fact)	6	0.741	+0.022
raw 3, 4x the trees	3	0.617	+0.005

3 engineered columns

+0.129 R²

4x the compute, no new columns

+0.005 R²

Features beat capacity, by a factor of 25 here

Why each one helped

kg_per_km: a ratio; a tree needs many splits to approximate a division

over_legal_hours: a legal cliff at 9h that no smooth function would find

is_weekend: a fact about the calendar, absent from every raw column

none of the three is derivable from more trees or more depth

Why this matters

The last row is the control. Quadrupling the model's budget on the raw columns buys 0.005; three lines of domain knowledge buy 0.129.

Common misconception

Wrong idea: modern models learn interactions on their own, so feature engineering is legacy practice.

Correct idea: they learn interactions they can represent. Ratios, thresholds, cyclical structure and information from outside the table are not among them.

import pandas as pd, numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import HistGradientBoostingRegressor

base = ["distance_km", "weight_kg", "driver_hours"]
cross_val_score(HistGradientBoostingRegressor(), df[base], y, cv=5).mean()   # 0.612

# three columns from domain knowledge, not from the algorithm
df["kg_per_km"] = df["weight_kg"] / df["distance_km"].clip(lower=0.1)
df["over_legal_hours"] = (df["driver_hours"] > 9).astype(int)
df["is_weekend"] = df["pickup_ts"].dt.dayofweek.isin([5, 6]).astype(int)

cross_val_score(HistGradientBoostingRegressor(),
                df[base + ["kg_per_km", "over_legal_hours", "is_weekend"]],
                y, cv=5).mean()                                              # 0.741

Generating hundreds of candidate features automatically and keeping whatever improves the validation score. With enough candidates some will improve it by chance, and you have just fitted the validation set — the selection-bias problem from the Cross-Validation module, at feature scale. Keep features you can justify, and confirm the ones you cannot on a set you have not touched.

Log and power transforms for skewed features

What it is

log1p(x) compresses a long right tail, turning a multiplicative spread into an additive one; sqrt is gentler, and PowerTransformer (Yeo–Johnson) fits the exponent and handles negatives and zeros. All are monotone, so they never reorder the values.

Why it matters

Two reasons that apply to different models. For linear and distance-based models it makes the feature's influence roughly proportional rather than dominated by a handful of extreme rows. For any model it turns a multiplicative relationship into an additive one, which is what a sum of terms can actually express.

In the real world: Customer spend ranges from ₹100 to ₹10 lakh; a log makes it usable for models that expect evenly spread values.

Visual example

Skew 4.8 -> 0.3

The same five incomes, raw and logged. The gaps become comparable instead of one row owning the whole range.

income (raw)

	income	gap to previous
p10	18000	-
p50	46000	28000
p90	112000	66000
p99	410000	298000
max	3800000	3390000

np.log1p(income)

log1p(income)

	log1p	gap to previous
p10	9.8	-
p50	10.74	0.94
p90	11.63	0.89
p99	12.92	1.29
max	15.15	2.23

What the compression does

raw: the max is 211x the p10, and the top gap is 121x the bottom gap

logged: the max is 1.5x the p10, and the top gap is 2.4x the bottom gap

a linear coefficient now means 'per 1% change', not 'per dollar'

a Euclidean distance is no longer decided entirely by the one 3.8M row

Why this matters

The gap column is the point. In raw units the single largest row accounts for most of the feature's total variation, so in any squared-error or distance calculation it is most of the feature.

Common misconception

Wrong idea: a log transform removes outliers.

Correct idea: it is monotone, so the order and the identity of every row are preserved exactly. It changes the spacing, not the membership.

from sklearn.preprocessing import PowerTransformer

df["income"].skew()                     # 4.8 — a long right tail
np.log1p(df["income"]).skew()           # 0.3 — near symmetric

# log1p, not log: it maps 0 -> 0 rather than -inf
np.log1p(0), np.log1p(np.array([0, 10, 1000]))

pt = PowerTransformer(method="yeo-johnson", standardize=True)
X_t = pt.fit_transform(X_train)         # fits lambda per column, on train only
pt.lambdas_.round(3)

Applying it to a tree-based model and expecting a change. A tree only compares a feature to a threshold, so any monotone transform leaves it bit-for-bit identical — the log of the threshold is the threshold of the log. Log transforms are for linear models, KNN, SVMs and neural networks, not for forests or GBMs.

Binning continuous features

What it is

Replacing a continuous column with an ordered set of intervals: pd.cut for fixed edges, pd.qcut or KBinsDiscretizer(strategy='quantile') for equal-frequency bins. The output is then treated as ordinal or one-hot encoded.

Why it matters

It lets a linear model fit a step function — one coefficient per bin — so a genuinely non-monotone relationship such as risk against age becomes representable. It also encodes real-world cliffs directly: a legal threshold, a pricing tier, a retirement age.

In the real world: Turning age into bands like 18–25 and 26–35 for a marketing model the team can explain.

Visual example

pd.cut

A non-monotone risk curve. A single linear coefficient must average it to nearly flat; five bins reproduce the shape.

True rate by ageBinned model (5 bins)Single linear term

Why one coefficient cannot work

rate falls from 0.158 at 20 to 0.049 at 45, then rises to 0.128 at 75

the best single slope is -0.0002 per year: effectively flat, R² ~ 0.01

5 bins -> 5 coefficients -> the U is captured, R² ~ 0.87

cost: within the 30-44 bin, ages 30 and 44 are now indistinguishable

Why this matters

Binning trades resolution for shape. Where the relationship is non-monotone, shape is worth far more than resolution — and where it is monotone, binning is pure loss.

Common misconception

Wrong idea: binning is a way to handle outliers and noise in a numeric column.

Correct idea: its real purpose is to let an additive model express a non-monotone relationship. For a tree it removes thresholds the tree would have found by itself.

df["age_band"] = pd.cut(df["age"], bins=[0, 18, 30, 45, 65, 120],
                        labels=["minor", "18-29", "30-44", "45-64", "65+"])

# data-driven, equal-frequency bins:
df["income_decile"] = pd.qcut(df["income"], q=10, labels=False, duplicates="drop")

from sklearn.preprocessing import KBinsDiscretizer
KBinsDiscretizer(n_bins=8, encode="onehot-dense", strategy="quantile")

Choosing the bin edges by looking at the target. Edges picked to maximise the difference in mean y between bins are fitted parameters, so they leak — put the discretiser inside the pipeline. And binning always destroys information: within a bin, 30 and 44 become identical. For a tree it is strictly harmful, since a tree finds better thresholds itself.

Practise it: How Wrong Are Your Probabilities?

Interaction and polynomial features

What it is

PolynomialFeatures(degree=2) appends every square and every pairwise product of the input columns; interaction_only=True appends the products alone. A linear model on those columns is linear in the parameters and nonlinear in the original features.

Why it matters

It is how a linear model represents 'the effect of dose depends on weight'. Additivity is the linear model's core limitation, and a product term is the minimal repair. The same features also help a tree with the diagonal-boundary problem from module 9.

In the real world: “Discount × holiday” can matter more to sales than either on its own.

Visual example

PolynomialFeatures

Column count against degree and input width. Degree 3 on 20 features is 1,770 columns — more than most datasets have rows.

interaction_only=True

drops the pure powers: usually the cheap win

n_output_features_

input cols	degree 2	degree 2 (interaction only)	degree 3
5	20	10	55
10	65	45	285
20	230	190	1770
58	1769	1653	34279

degree=3

cubic growth — almost never worth it

Always pair an expansion with regularisation

The formula

degree 2 adds d pure squares plus d(d-1)/2 pairwise products

d = 10: 10 + 10 + 45 = 65 output columns

interaction_only=True: 10 + 45 = 55, minus the bias -> 45 new

d = 58: 1769 columns from a 569-row dataset. Ridge is not optional.

Why this matters

The highlighted cells are where the expansion has more columns than a typical dataset has rows, at which point an unregularised linear model can fit it perfectly and has learnt nothing.

Common misconception

Wrong idea: polynomial features are a general way to make a linear model nonlinear.

Correct idea: they are a quadratically expensive way, usable on few features. Past a dozen columns, a tree ensemble finds interactions more cheaply than you can enumerate them.

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import RidgeCV

pipe = make_pipeline(
    PolynomialFeatures(degree=2, include_bias=False),
    StandardScaler(),          # AFTER expansion: x^2 has a very different scale
    RidgeCV(alphas=np.logspace(-3, 3, 13)),
)
cross_val_score(pipe, X, y, cv=5, scoring="r2").mean()

PolynomialFeatures(2).fit(np.zeros((1, 10))).n_output_features_   # 65 from 10
PolynomialFeatures(3).fit(np.zeros((1, 10))).n_output_features_   # 285

The count explodes. Degree 2 on d features gives d(d+3)/2 columns — 65 from 10, 1,769 from 58 — and degree 3 is cubic. Beyond a handful of features you must either restrict to interaction_only, name the interactions you actually believe in, or let a tree ensemble find them (a depth-d tree represents up to d-way interactions for free).

Date and time features

What it is

A timestamp is not a number your model can use. Extract the components that carry signal — hour, day of week, month, is-weekend, is-holiday, days-since-signup, days-until-renewal — and drop the raw timestamp, which is a monotone counter the model will happily overfit.

Why it matters

Nearly every behavioural target has calendar structure, and none of it is visible in a Unix epoch. Duration features (days_since_last_order) are usually stronger than any single component, because they describe the row's position in *its own* history rather than in the calendar.

In the real world: Day of week, month-end and paydays explain big swings in orders and card spending.

ts = df["pickup_ts"]
df["hour"] = ts.dt.hour
df["dow"] = ts.dt.dayofweek
df["is_weekend"] = df["dow"].isin([5, 6]).astype(int)
df["month"] = ts.dt.month
df["days_since_signup"] = (ts - df["signup_ts"]).dt.days
df["is_month_end"] = ts.dt.is_month_end.astype(int)

# drop the raw timestamp: as an integer it is a perfect row-order proxy
X = df.drop(columns=["pickup_ts", "signup_ts"])

Leaving the raw timestamp in as a numeric column. A tree splits on it and effectively memorises 'rows before March behaved like this', which validates beautifully on a shuffled split and fails completely on future data — and it is exactly why the Cross-Validation module insisted on TimeSeriesSplit for ordered data.

Cyclical encoding with sine and cosine

What it is

Periodic components have no linear order: hour 23 and hour 0 are one hour apart but 23 units apart as integers. Encode them as a pair, sin(2*pi*h/24) and cos(2*pi*h/24), which places the 24 hours evenly around a unit circle so that adjacent hours are adjacent points.

Why it matters

It makes the distance between two times correct for every model that measures distance or fits a coefficient. Without it, the midnight boundary is an artificial cliff: a linear model must average across it, and a tree needs a separate split on each side to describe one continuous pattern.

In the real world: Telling a model that 11 pm and 1 am are close in time, for predicting ride demand around midnight.

Visual example

Cyclical encoding

All 24 hours as (sin, cos) pairs. They form a circle, so hour 23 sits right beside hour 0 — which the raw integers get maximally wrong.

Night (21-02)Morning (03-08)Afternoon (09-14)Evening (15-20)

Distances, three ways

hours 23 and 0: raw |23 - 0| = 23 (max), encoded distance 0.261 (min)

hours 11 and 13: raw 2, encoded 0.518 — 2 steps around the circle

hours 0 and 12: raw 12, encoded 2.000 — the diameter, genuinely opposite

every adjacent pair is exactly 2 sin(pi/24) = 0.261 apart, all the way round

Why this matters

The gap between hour 23 and hour 0 in the picture is identical to the gap between hour 3 and hour 4. That uniformity is the property the raw integer column does not have anywhere except in its interior.

Common misconception

Wrong idea: sin/cos encoding is a normalisation of the hour column.

Correct idea: it is a change of topology: a line becomes a circle. One column that wraps is replaced by two that do not need to.

1Pick the period24 for hours, 7 for weekdays, 12 for months, 365.25 for day-of-year.
2Emit bothsin(2πx/p) and cos(2πx/p) — one alone is ambiguous.
3Check a wrapConfirm the last value and the first are now adjacent.

h = df["hour"].to_numpy()
df["hour_sin"] = np.sin(2 * np.pi * h / 24)
df["hour_cos"] = np.cos(2 * np.pi * h / 24)

# the whole point, checked:
def enc(x, period=24):
    return np.c_[np.sin(2*np.pi*x/period), np.cos(2*np.pi*x/period)]

np.linalg.norm(enc(23) - enc(0))     # 0.261 — adjacent, as it should be
np.linalg.norm(enc(23) - enc(11))    # 1.932 — genuinely far apart
abs(23 - 0), abs(23 - 11)            # (23, 12) — the raw integers, backwards

# same pattern for day-of-week (period 7) and month (period 12)

Two things. Both columns are needed — sin alone maps hour 6 and hour 18 to the same value, so the encoding is not injective. And the pair is a *rotation*, so a tree still has to cut it with axis-aligned splits and gains less from it than a linear model, KNN or an SVM does; for a GBM, plain integer hour plus an explicit is_night flag is often just as good.

Aggregation features from groups and windows

What it is

Summarise each row against a group it belongs to, or against its own recent past: a per-user mean, a count, a deviation from the group mean, a 7-day rolling average. This is exactly Pandas' groupby().transform() and rolling(), used as a feature-construction step.

Why it matters

It gives the model context that a single row cannot contain. 'This order is £200' is weak; 'this order is 4.1x this customer's median order' is strong, and no model can compute the second from the first because the customer's history is not in the row.

In the real world: A customer's average spend over the last 30 days, compared with today's purchase, is a classic fraud signal.

Visual example

groupby().transform()

transform returns a value per row, not per group — which is what makes an aggregate usable as a feature.

user	amount
u1	50
u2	210
u1	30
u2	190
u1	205

split

user	amount
u1	50
u1	30
u1	205

user	amount
u2	210
u2	190

combine

df with the new features

user	amount	user_mean	amount_vs_user
u1	50	95	0.53
u2	210	200	1.05
u1	30	95	0.32
u2	190	200	0.95
u1	205	95	2.16

The last row, before and after

u1's amounts are 50, 30, 205 -> mean 95

row 5: amount 205, user_mean 95 -> ratio 2.16

row 2: amount 210, user_mean 200 -> ratio 1.05

as raw amounts 205 and 210 are near-identical; as ratios they are 2.16 vs 1.05

Why this matters

Rows 2 and 5 have almost the same amount and completely different meanings. The ratio column is the one that separates them, and it is arithmetically unavailable from a single row.

Common misconception

Wrong idea: agg and transform are two ways to compute the same thing.

Correct idea: agg collapses each group to one row; transform broadcasts the group's value back to every member. Only the second shape can be a feature column.

g = df.groupby("user_id")["amount"]
df["user_mean"] = g.transform("mean")        # broadcast back to every row
df["user_count"] = g.transform("count")
df["amount_vs_user"] = df["amount"] / df["user_mean"]
df["amount_z_in_user"] = (df["amount"] - g.transform("mean")) / g.transform("std")

# time-aware: only the past, and excluding the current row
df = df.sort_values("ts")
df["roll7"] = (df.groupby("user_id")["amount"]
                 .transform(lambda s: s.shift(1).rolling(7, min_periods=1).mean()))

Aggregating over the full dataset when the target is time-ordered. A plain groupby().transform("mean") includes the row itself and every later row, so the feature contains the future — the .shift(1) before .rolling() is what makes it honest. And the aggregate must be computed from training rows only, or from a strictly prior window, or you have leaked twice over.

Target encoding, done out of fold

What it is

Replace a high-cardinality category with a statistic of the target within that category — usually the mean, smoothed toward the global mean for rare levels. The statistic must be computed from rows *other than the one being encoded*: for each fold, encode it using only the other folds' target values.

Why it matters

It compresses thousands of levels into one informative numeric column, where one-hot encoding would produce thousands of sparse columns. Done in fold, it is one of the strongest features available on categorical tabular data; done naively, it is the most reliable way to produce a perfect validation score and a worthless model.

In the real world: Using each merchant's past fraud rate as a feature without leaking the answer into training.

Visual example

TargetEncoder(cv=5)

Five rows, one rare city. The naive encoding of row 5 is its own label; the out-of-fold encoding cannot see it.

Seen once

(naive encoding = its own y)

naive

corr(feature, y) = 1.0 — it IS y

city -> encoded value

	city	y	naive mean	out-of-fold
r1	London	1	0.667	0.5
r2	London	1	0.667	0.5
r3	London	0	0.667	1
r4	Leeds	0	0	0.4
r5	Hull	1	1	0.4

out-of-fold

falls back to the global mean, 0.4

Naive AUC 1.000 in validation, ~0.50 in production

Row 5, both ways

naive: mean of y over all Hull rows = mean([1]) = 1.0 — that row's own label

out-of-fold: Hull does not appear in the other folds, so fall back to mean(y) = 0.4

row 3 (London, y=0): naive 0.667, out-of-fold mean([1, 1]) = 1.0

smoothing with m=10: (3 x 0.667 + 10 x 0.4) / 13 = 0.462 — closer, but still in-fold

Why this matters

The naive column correlates perfectly with y on the training rows and not at all on new ones. That is the exact signature of leakage, and it is invisible in any metric computed on the same rows.

Common misconception

Wrong idea: smoothing the category means is enough to make target encoding safe.

Correct idea: smoothing reduces the leak's size; only computing the statistic from other rows removes it. Use cross-fitting, and pass your groups through if the data is grouped.

1SplitPartition the training rows into cv folds.
2Encode out of foldEach fold's rows are encoded from the other folds' targets only.
3Refit for transformAt predict time, use the statistic from all the training rows.

from sklearn.preprocessing import TargetEncoder
from sklearn.pipeline import make_pipeline

# scikit-learn's TargetEncoder fits with internal cross-fitting
enc = TargetEncoder(target_type="binary", smooth="auto", cv=5)
pipe = make_pipeline(enc, HistGradientBoostingClassifier())
cross_val_score(pipe, X_cat, y, cv=cv, scoring="roc_auc").mean()

# the leaky version, for contrast — do not ship this
means = df.groupby("city")["y"].mean()
df["city_te"] = df["city"].map(means)     # every row encoded using its own y

The leak is at its worst exactly where the encoding is most tempting. For a category appearing once, the naive mean *is* that row's label, so the column is a copy of y — validation AUC near 1.0, production performance at chance. Smoothing reduces it; only out-of-fold computation removes it.

Text features: bag-of-words and TF-IDF

What it is

CountVectorizer turns a document into a sparse vector of term counts over a fixed vocabulary; TfidfVectorizer scales each count by the inverse document frequency of its term, so words that appear in every document count for little. ngram_range=(1, 2) adds adjacent word pairs, recovering a little word order.

Why it matters

It makes text usable by every model in this syllabus with no deep-learning machinery. A TF-IDF matrix with a linear SVM or MultinomialNB remains a strong, fast, interpretable baseline for topic and sentiment classification, and it is the thing any transformer has to beat to be worth its cost.

In the real world: Turning product reviews into numbers to predict star ratings or spot complaints.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC

vec = TfidfVectorizer(min_df=3, max_df=0.9, ngram_range=(1, 2),
                      sublinear_tf=True, strip_accents="unicode")
clf = make_pipeline(vec, LinearSVC(C=1.0))
cross_val_score(clf, docs, labels, cv=5, scoring="f1_macro").mean()

clf.fit(docs, labels)
X_t = clf[0].transform(docs)
X_t.shape, X_t.nnz / np.prod(X_t.shape)     # ~0.001 dense: keep it sparse

Fitting the vectoriser on all the documents before splitting. The vocabulary and the IDF weights are both fitted from the corpus, so a vectoriser fitted on everything has read the test documents — the same Pipeline rule as every other fitted transform. Also resist .toarray(): a 50k x 200k sparse matrix is 40MB sparse and 80GB dense.

The discipline: hypothesise, build, validate, keep or discard

What it is

A working loop. Start from a domain hypothesis about what should matter; build the smallest feature that encodes it; measure it against the current model with the same CV protocol; keep it only if the gain exceeds the fold-to-fold noise; and write down why it is there.

Why it matters

It is what separates feature engineering from feature accumulation. A feature you cannot explain is a feature nobody can debug when it drifts, and a feature kept on a gain smaller than the CV standard deviation is noise you have promoted to a column.

In the real world: Adding one new feature at a time to the churn model and keeping it only if validation improves.

Visual example

Candidate log

A candidate log, scored with paired per-fold differences. Two of the five gains are smaller than their own noise.

delta ROC-AUC vs the current model

candidate	hypothesis	delta	std	verdict
kg_per_km	density drives cost	+0.0181	0.0042	keep
hour_sin/cos	demand is cyclical	+0.0094	0.0031	keep
user_mean_amount	context beats level	+0.0129	0.0038	keep
distance^2	diminishing returns	+0.0011	0.0029	drop
raw pickup_ts	(none — a counter)	+0.0240	0.018	drop: leak

delta < 2 x se

indistinguishable from zero

big gain, no hypothesis

the classic leak signature

3 kept, 2 dropped — one for being noise, one for being a leak

The two rejections

distance^2: +0.0011 with std 0.0029, se = 0.0013 -> inside the noise

raw pickup_ts: the largest gain AND the largest variance, with no hypothesis

a timestamp as an integer is a row-order proxy: it wins on a shuffled split only

re-scored under TimeSeriesSplit it is -0.0310: the leak, exposed

Why this matters

The biggest number in the delta column is the one to reject. A large, unexplained, high-variance gain is what leakage looks like from the outside, every time.

Common misconception

Wrong idea: a feature that improves the CV score is a good feature.

Correct idea: a feature is good when the gain exceeds its own noise and you can say why it should exist. Without the second, the first is often a leak you have not found yet.

1HypothesiseSay in words what the feature should capture, before building it.
2Measure pairedSame folds, per-fold differences against the current model.
3Keep or discardKeep only if the mean gain clears roughly two standard errors.

base = cross_val_score(model, X, y, cv=cv, scoring="roc_auc", n_jobs=-1)

def try_feature(name, col):
    s = cross_val_score(model, np.c_[X, col], y, cv=cv,
                        scoring="roc_auc", n_jobs=-1)
    delta = s - base                    # PAIRED, fold by fold
    verdict = "keep" if delta.mean() > 2 * delta.std() / np.sqrt(len(delta)) else "drop"
    print(f"{name:22s} {delta.mean():+.4f} +/- {delta.std():.4f}  {verdict}")

try_feature("kg_per_km", df["weight_kg"] / df["distance_km"])
try_feature("hour_sin", np.sin(2 * np.pi * df["hour"] / 24))

Evaluating candidates one at a time and keeping everyone that helps. Features interact: two ratios that each add 0.01 alone may add 0.01 together because they encode the same thing. Re-measure the set you are keeping as a set, and keep a final untouched holdout to confirm the whole accumulated story once, at the end.