Machine Learning · Module 3 of 26 · 10 concepts · about 17 min

Preparing Data for ML

Splitting first, then imputing, scaling and encoding with parameters learned from the training rows only — and the leakage that follows from doing it in any other order.

At work: Getting real company data ready for a model without accidentally letting it cheat, the most common reason models fail after launch.

Split first: the train/test split comes before every statistic

What it is

The very first operation on a dataset is holding rows out. Only then may anything be computed from the data — a mean, a standard deviation, a category list, a quantile, a correlation used to choose features.

Why it matters

Any number computed before the split has seen the held-out rows, so the held-out score is no longer an estimate of performance on unseen data. It is the one error that makes every subsequent measurement in the project untrustworthy.

In the real world: Computing average income on all customers before splitting lets test data leak into training and makes a credit model look better than it is.

Visual example

Leakage

The mean used to transform the test rows must come from train rows only — and it is a different number.

WRONG — mean over all 5 rows

	value	centred
t0	10	-14
t1	20	-4
t2	30	6
e0	40	16
e1	20	-4

split, then fit the scaler

RIGHT — mean over the 3 train rows

	value	centred
t0	10	-10
t1	20	0
t2	30	10
e0	40	20
e1	20	0

Where the two means come from

all five rows: (10+20+30+40+20)/5 = 24 <- leaked

train rows only: (10+20+30)/3 = 20 <- correct

e0 centred: 40-24 = 16 (leaked) vs 40-20 = 20 (correct)

Why this matters

It shows leakage as a concrete numerical difference rather than a rule to obey. The test rows influenced their own transformation, which production data can never do.

Common misconception

Wrong idea: scaling is harmless preprocessing, so the order does not matter.

Correct idea: a scaler is fitted: it has learned parameters. Learning them from all the data is training on the test set, exactly as much as fitting the model on it would be.

Same data, same transformer, different parameters — because the left-hand mean was allowed to see e0 and e1.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0, stratify=y
)

scaler = StandardScaler().fit(X_train)     # statistics from train rows only
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)        # transform, never fit_transform

X_scaled = StandardScaler().fit_transform(X) followed by a split. It looks tidier and it is the most common leak in real notebooks: the mean and standard deviation used on the test rows were computed partly from the test rows.

Practise it: A Split That Does Not Leak Across Groups

Fit on train, transform everything: the leakage rule

What it is

Every transformer follows the same two-call discipline. fit learns parameters — means, scales, category lists, quantile boundaries — from the training rows; transform applies those stored parameters to any rows, including test rows and, later, live traffic.

Why it matters

It mirrors production exactly. At serving time a single request arrives with no distribution of its own, so it must be transformed using parameters frozen at training time. Any offline step that cannot be reproduced that way is measuring an impossible system.

In the real world: The scaling used on live loan applications must be the one learned from training data, exactly as it will be in production.

Visual example

Transformer API

fit learns numbers; transform applies them. **Only the training rows may reach fit.**

The whole rule

(one fit, many transforms)

Who may call what

	fit()	transform()	why
X_train	yes	yes	the only rows the model may learn from
X_val	no	yes	must be scored as unseen data
X_test	no	yes	must be scored as unseen data
one live row	no	yes	has no distribution of its own

scaler.transform(row)

frozen mean_ and scale_ applied

fit_transform is legal on X_train only

Standardising one test value with train parameters

scaler.mean_ = 20.0, scaler.scale_ = 8.16 (from train)

test value 40 -> (40 - 20.0) / 8.16 = 2.45

a refit on test would give a different, flattering number

Why this matters

It is the single habit that keeps offline numbers comparable to live ones, and it applies unchanged to imputers, encoders, feature selectors and dimensionality reduction.

Common misconception

Wrong idea: fit_transform is just the shorthand, so it is fine anywhere.

Correct idea: it is fit plus transform. Using it on validation or test data silently relearns the parameters and destroys the held-out guarantee.

scaler = StandardScaler()
scaler.fit(X_train)
scaler.mean_, scaler.scale_        # the frozen parameters, shipped with the model

X_test_s = scaler.transform(X_test)

# One live row, months later, transformed with the same frozen numbers:
scaler.transform([[41.0, 82_000.0]])

Calling fit_transform on the test set. It refits, so the test rows get their own mean and their own scale — a different feature space from the one the model was trained in. The score usually still looks plausible, which is why it survives review.

Missing-data imputation strategies

What it is

Filling gaps with a constant learned from the training rows: the column mean (symmetric numeric data), the median (skewed or outlier-heavy), the most frequent value (categoricals), or an explicit sentinel category for 'unknown'.

Why it matters

Most estimators refuse to fit with NaN present, and dropping rows throws away everything else those rows contained. A stated imputation rule also makes the treatment reproducible at serving time, where a missing field is routine.

In the real world: Filling missing incomes in loan applications with the median, instead of dropping those customers, keeps the model usable for everyone.

Visual example

Imputation

Mean and median give different fills on skewed data, and the flag preserves the fact that a value was missing.

income, with gaps

	income
0	30
1	34
2	NaN
3	38
4	220

SimpleImputer + missingness flag

filled, plus was_missing

	mean-filled	median-filled	was_missing
0	30	30	0
1	34	34	0
2	80.5	34	1
3	38	38	0
4	220	220	0

The two fill values

mean of present values: (30+34+38+220)/4 = 80.5

median of present values: (34+38)/2 = 36 -> 34 with train rows only

add_indicator=True keeps 'this was missing' as its own feature

Why this matters

Imputation invents data. Choosing the statistic deliberately, and flagging where you invented it, is what keeps the invention visible to the model and to you.

Common misconception

Wrong idea: imputation is a neutral cleanup step.

Correct idea: it adds a fabricated value at every gap. Which statistic you fabricate with changes the fitted model, and the missingness itself is often predictive — so keep the flag.

One outlier of 220 drags the mean fill to 80.5 — a value unlike anything else in the column. The median fill of 34 sits where the data is.

from sklearn.impute import SimpleImputer

num = SimpleImputer(strategy="median").fit(X_train[num_cols])
cat = SimpleImputer(strategy="most_frequent").fit(X_train[cat_cols])
num.statistics_          # the learned fill value per column

X_train[num_cols] = num.transform(X_train[num_cols])
X_test[num_cols] = num.transform(X_test[num_cols])

Imputing with the mean of the whole column, computed before the split. It is the leakage rule again, and it is easy to miss because fillna(df["x"].mean()) reads so naturally in Pandas.

Practise it: A Pipeline That Does Not Leak

The missingness indicator

What it is

A companion binary column per imputed feature, recording whether the original value was absent. SimpleImputer(add_indicator=True) and MissingIndicator generate them.

Why it matters

Missingness is frequently informative: an unfilled income field may mean a self-employed applicant, an absent last-login may mean a brand-new account. Imputing without the flag erases that signal permanently.

In the real world: Whether a customer left their phone number blank can itself predict fraud; a flag column keeps that signal.

from sklearn.impute import SimpleImputer

imp = SimpleImputer(strategy="median", add_indicator=True).fit(X_train)
imp.transform(X_train).shape      # original columns + one flag per gappy column
imp.indicator_.features_          # which column indices got a flag

Assuming missingness is random. If a field stops being collected halfway through your history, the flag becomes a proxy for 'recent row' and the model learns the date rather than the behaviour — a leak dressed as a feature.

Scaling: StandardScaler, MinMaxScaler, RobustScaler

What it is

StandardScaler subtracts the mean and divides by the standard deviation. MinMaxScaler maps the observed range onto [0, 1]. RobustScaler subtracts the median and divides by the interquartile range, so extreme values barely move the parameters.

Why it matters

Distance-based models (k-NN, SVM, k-means), any penalised linear model, and gradient descent itself are all scale-sensitive: an unscaled feature measured in tens of thousands dominates one measured in single digits regardless of its relevance.

In the real world: Income in lakhs and age in years sit on very different scales; scaling stops income from dominating a distance-based model.

Visual example

Scaling

One outlier breaks MinMax and barely moves Robust. Follow each line over the 20-45 range where the real data sits.

StandardScalerMinMaxScalerRobustScaler

Parameters, and the span they give the real data

Standard: mean 80, sd 80 -> band 20..45 maps to -0.75..-0.44

MinMax: min 20, max 220 -> band maps to 0.00..0.10

Robust: median 36, IQR 14 -> band maps to -1.14..+0.64

Why this matters

It makes the choice mechanical: if the column has heavy tails, the scaler whose parameters are outlier-driven (min/max, and to a lesser degree mean/sd) destroys the resolution where the data actually is.

Common misconception

Wrong idea: MinMaxScaler is the safe choice because the output is bounded in [0, 1].

Correct idea: bounded is not informative. With one extreme value, the bound is spent on the outlier and the ordinary rows become indistinguishable. Use the median and IQR when tails are heavy.

Inside the shaded band MinMax spans 0.10 of its range, Standard spans 0.31, Robust spans 1.43. The wider the spread there, the more the model can tell those rows apart.

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

for scaler in [StandardScaler(), MinMaxScaler(), RobustScaler()]:
    z = scaler.fit(X_train[["income"]]).transform(X_test[["income"]])
    print(type(scaler).__name__, z.min().round(2), z.max().round(2))

# Trees do not need any of this: a split threshold is scale-invariant.

Using MinMaxScaler on a column with outliers. A single value of 1,000,000 pins the maximum, and the 99% of ordinary values get squeezed into the first few thousandths of the [0, 1] range, where the model can no longer distinguish them.

Encoding categoricals: one-hot and ordinal

What it is

One-hot encoding creates one binary column per category, so no order is implied. Ordinal encoding maps categories to integers, which implies an order and a spacing — appropriate only when one genuinely exists.

Why it matters

A linear model reads an ordinally encoded city column as a number, so city=3 is treated as three times city=1. That is meaningless and it does affect the coefficients. One-hot avoids the implication at the cost of width.

In the real world: Turning “city” into columns a model can use, and “small/medium/large” into ordered numbers.

Visual example

Encoding

One-hot says 'different'; ordinal says 'ordered and evenly spaced' — a claim about cities that is simply false.

city (3 categories)

	city	ordinal
0	leeds	0
1	derby	1
2	york	2
3	leeds	0

OneHotEncoder vs OrdinalEncoder

one-hot: no order implied

	c_leeds	c_derby	c_york
0	1	0	0
1	0	1	0
2	0	0	1
3	1	0	0

What the ordinal column tells a linear model

york (2) - derby (1) = derby (1) - leeds (0): equal steps, invented

coefficient b means 'each unit of city adds b' — meaningless here

one-hot gives each city its own coefficient, which is the intent

Why this matters

It is the difference between a model that learns three city effects and one that learns a single slope through an arbitrary alphabetical ordering.

Common misconception

Wrong idea: ordinal encoding is fine because the model will work it out.

Correct idea: a linear model cannot work it out — it only has one coefficient for the column. Tree models can, by splitting repeatedly, which is why the answer depends on the model family.

An unseen city transforms to 0, 0, 0 with handle_unknown="ignore" — a legal row rather than an exception.

from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

ohe = OneHotEncoder(handle_unknown="ignore", sparse_output=False).fit(X_train[cat_cols])
ohe.categories_            # the vocabulary, learned from train only
ohe.transform(X_test[cat_cols])

# Genuine order: state it, do not let the encoder guess.
OrdinalEncoder(categories=[["small", "medium", "large"]]).fit(X_train[["size"]])

Leaving handle_unknown at its default. A category that appears only in the test set (or in live traffic) then raises at transform time, which is a production outage rather than a modelling issue. "ignore" encodes it as all-zeros instead.

Target encoding, and its specific leakage risk

What it is

Replacing a category with a statistic of the target computed within that category — usually its mean. It keeps the column narrow, which matters when a categorical has thousands of levels.

Why it matters

High-cardinality columns (postcode, product id, merchant) are impractical to one-hot and highly predictive. Target encoding is the standard compromise, and it is the one encoder that reads the label, so it carries a leakage mode all of its own.

In the real world: Replacing each of 20,000 pincodes with its average fraud rate is powerful, and leaks badly unless done on out-of-fold data.

Visual example

Target encoding

A rare category's own label leaks into its encoding. Out-of-fold means and smoothing are the fix.

A copy of y

(one row, one label)

mean target per category

	naive mean	leaks own label?	smoothed
A (40 rows)	0.31	barely	0.31
B (12 rows)	0.58	some	0.49
C (1 row)	1	entirely	0.34

Smoothing pulls small categories toward the global mean, 0.33

Smoothing one rare category

global mean = 0.33, smoothing weight m = 20

C: (1 x 1.0 + 20 x 0.33) / (1 + 20) = 0.36

A: (40 x 0.31 + 20 x 0.33) / (40 + 20) = 0.32 — barely moved

Why this matters

It shows why target encoding needs machinery that one-hot does not: the encoder has read the label, so without out-of-fold computation the feature is partly the answer.

Common misconception

Wrong idea: target encoding leaks only if you fit it on the test set.

Correct idea: it leaks within the training set too. A row's own target inside its category mean is leakage into that row's features, which is what out-of-fold encoding removes.

from sklearn.preprocessing import TargetEncoder

# cv>1 makes the encoding of each training row use folds that exclude it.
enc = TargetEncoder(smooth="auto", cv=5).fit(X_train[cat_cols], y_train)
X_train_e = enc.transform(X_train[cat_cols])
X_test_e = enc.transform(X_test[cat_cols])       # train-fitted means
enc.encodings_[0][:5]

Encoding a category with a mean that includes the row being encoded. For a category with one row, the encoded value *is* that row's label — a perfect feature offline and pure noise live. Use out-of-fold encoding and smoothing toward the global mean.

Outlier handling

What it is

Deciding, per column, whether extreme values are errors (a negative age, a £10 million grocery order), genuine rare events, or heavy-tailed normality — and treating each differently: correct, keep, clip, or transform.

Why it matters

Outliers move squared-error fits, minimum/maximum scalers and means far more than their count suggests. Deleting them makes a tidier model of a world that still contains them; keeping them unexamined lets a data-entry error set your scaling.

In the real world: A few ₹50 lakh orders from corporate buyers can distort a model built for everyday shoppers.

Visual example

Outliers

One extreme row changes the mean by 38 and the median by 2. Which statistic you use decides how much it matters.

Outlier-driven

(mean and std move hugely)

amount, with and without one row

	with outlier	without
mean	80.5	42
median	36	34
std	80.2	4.1
max	220	40

Median and IQR are the stable pair — hence RobustScaler

Why this matters

It connects outlier handling to the scaler choice above: the statistics an outlier can capture are exactly the ones you should avoid depending on when the tail is heavy.

Common misconception

Wrong idea: outliers should be removed before modelling.

Correct idea: only errors should be removed. Real extremes are data; clip them, transform them, or use an outlier-insensitive loss — but do not pretend the population lacks them.

import numpy as np

lo, hi = np.percentile(X_train["amount"], [1, 99])     # bounds from train only
X_train["amount_c"] = X_train["amount"].clip(lo, hi)
X_test["amount_c"] = X_test["amount"].clip(lo, hi)

X_train["amount_log"] = np.log1p(X_train["amount"])    # tame a heavy tail instead

Dropping rows outside three standard deviations as routine hygiene. On a heavy-tailed column that removes a real and often valuable population — the big spenders, the fraud cases — and the model then meets them for the first time in production.

Pipelines: making the leakage rule structural

What it is

Pipeline chains transformers and a final estimator into one object. Calling fit fits each step on the training fold only; predict applies the stored parameters. ColumnTransformer routes different column groups to different transformers.

Why it matters

It turns the discipline from a habit into a property of the code. Inside cross-validation the pipeline refits the scaler and encoder on each fold's training rows automatically, which is the only correct behaviour and almost impossible to hand-roll without slipping.

In the real world: Bundling cleaning, scaling and the model means the credit model in production prepares data exactly as it did in testing.

Visual example

Pipeline

Inside cross-validation, a pipeline refits preprocessing per fold. Preprocessing once, outside, leaks into all of them.

The 0.027 is fictional

(leaked fold statistics)

5-fold CV, same data, two arrangements

	scaler fitted on	mean CV score	honest?
Preprocess then CV	all training rows, once	0.908	no
CV over a Pipeline	each fold's train part	0.881	yes

Ship the pipeline: joblib.dump(pipe, ...) keeps preprocessing with the model

Why this matters

A pipeline fixes leakage, fixes train/serve skew, and makes the deployable artifact the same object you evaluated. Three problems, one abstraction.

Common misconception

Wrong idea: a pipeline is a convenience wrapper for chaining steps.

Correct idea: it is a correctness device. It is what makes 'fit on train only' true inside every fold of every search, where doing it by hand reliably fails.

1Declare the stepsImputer, scaler, encoder, estimator — in order.
2Fit onceEach step fits on the rows it is given, and no others.
3Search over itGridSearchCV(pipe, {"clf__C": [...]}) tunes without leaking.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

numeric = Pipeline([("impute", SimpleImputer(strategy="median")),
                    ("scale", StandardScaler())])
prep = ColumnTransformer([("num", numeric, num_cols),
                          ("cat", OneHotEncoder(handle_unknown="ignore"), cat_cols)])
pipe = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000))])

cross_val_score(pipe, X_train, y_train, cv=5)      # refit per fold, no leakage

Preprocessing the whole training set once and then cross-validating the bare estimator on the result. Every fold's validation rows helped compute the scaler's parameters, so all five fold scores are optimistic — the leak is invisible because no test data was involved at all.

Practise it: A Pipeline That Does Not Leak

Class imbalance: what to notice now

What it is

When one class is rare — fraud at 0.2%, churn at 3% — accuracy stops being informative and a random split can leave a fold with almost no positives. The immediate practices are stratify=y when splitting, StratifiedKFold for cross-validation, and never quoting bare accuracy.

Why it matters

Those three habits cost nothing and prevent the two most common early mistakes: a validation fold with four positive examples, and a project that celebrates 0.98 accuracy from a model that predicts the majority class every time.

In the real world: Only 1 in 500 card transactions is fraud; spotting that early changes how the model is trained and judged.

from sklearn.model_selection import train_test_split, StratifiedKFold

y.mean()      # look at the positive rate before anything else

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0, stratify=y    # same rate in both halves
)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

Rebalancing the data — oversampling or class weights — before splitting. Duplicated positive rows then appear on both sides of the split, so the test set contains copies of training rows. Resampling belongs inside the pipeline, after the split, on the training fold only.

Practise it: Precision, Recall and F1 by Hand, Pick a Threshold for an Imbalanced Classifier