Machine Learning · Module 21 of 26 · 9 concepts · about 16 min

ML Pipelines

Chaining preprocessing and a model into one fitted object, so the transformations that produced a score travel with it and cannot silently see the test set.

At work: Packaging every preparation step with the model, so what was tested is exactly what runs in production.

Pipeline

What it is

Pipeline([(name, step), ...]) composes any number of transformers and one final estimator into a single object. Calling fit fits each step in order, passing each one's output to the next; calling predict runs the same chain forward through the already-fitted steps.

Why it matters

It collapses a scattered sequence of notebook cells into one thing you can cross-validate, tune, pickle and deploy. Without it, 'the model' is a fitted estimator plus an oral history of what was done to the data first, and the oral history is what breaks.

In the real world: Bundling cleaning, encoding, scaling and the model into one object, so production runs exactly what was tested.

Visual example

sklearn Pipeline

One object, one fit: raw columns go in the left end, a prediction comes out the right end.

X_raw — what you actually have

	age	income	city
0	34	52000	Pune
1	NaN	61000	Delhi
2	51	NaN	Pune

Pipeline([('impute', ...), ('scale', ...), ('clf', ...)])

what the estimator sees

	age_z	income_z	city=Pune
0	-0.71	-0.99	1
1	0	0.5	0
2	0.71	0.49	1

Why this matters

The arrow in this picture is the artifact. Ship only the right-hand estimator and you have shipped a model that expects numbers it will never be sent.

Common misconception

Wrong idea: a pipeline is a convenience wrapper that saves a few lines of code.

Correct idea: it is the unit of reproducibility. The fitted imputation medians, scaling statistics and category vocabulary are state learnt from training data, and the pipeline is what keeps that state attached to the model that depends on it.

1imputeMissing age and income filled with the training median.
2scaleNumeric columns centred and divided by the training standard deviation.
3encodecity becomes one indicator column per category seen in training.
4clfThe final step fits on the matrix the three transformers produced.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ('scale', StandardScaler()),
    ('clf', LogisticRegression(max_iter=1000)),
])

pipe.fit(X_train, y_train)      # fits the scaler, then the model on scaled data
pipe.predict(X_test)            # scales with the TRAINING statistics, then predicts
pipe.named_steps['clf'].coef_   # the inner estimator is still reachable

Every step except the last must implement transform; the last one only needs fit/predict. Putting a classifier in the middle of a pipeline fails with an error about a missing transform, which reads as a bug in the classifier rather than as a step in the wrong slot.

Practise it: A Pipeline That Does Not Leak

The fit / transform contract

What it is

fit learns state from the data it is given and stores it on the transformer; transform applies that stored state without changing it. fit_transform does both in one call and is only ever correct on training data.

Why it matters

This two-method split is the entire mechanism by which a pipeline prevents leakage. The test set is only ever passed to transform, so no statistic computed from it can reach the model — not the mean, not the variance, not the category list, not the quantile boundaries.

In the real world: Every step learns only from training data, then applies the same rules to live data.

Visual example

fit vs transform

The scaler holds one set of statistics, learnt from training rows, and reuses them on everything else.

transform only

(never fitted on)

fit sees these

(mean 40, sd 10)

scaler.fit(X_train)

stores mean_ = 40, scale_ = 10

age, before scaling

	age
train 0	30
train 1	40
train 2	50
val 0	70

scaler.transform(X_val)

(70 - 40) / 10 = 3.0

The val row lands at 3.0 — visibly extreme, which is the truth about it

Both ways of scaling the val row

correct: transform: (70 - 40) / 10 = 3.0 — an outlier, and the model is told so

leaky: fit_transform on val alone: (70 - 70) / 0 = 0.0 — it becomes perfectly average

Difference: the leaky path erases the one fact the model most needed

Why this matters

Leakage is rarely dramatic. Here it does not corrupt a label or duplicate a row; it just relocates an extreme value to the centre of the distribution, and the validation score goes up.

Common misconception

Wrong idea: scaling the validation set with its own mean is fairer, because each set is treated the same way.

Correct idea: at serving time you will have one row, with no mean of its own. transform is the only operation that exists in production, so it is the only one that may be used in evaluation.

scaler = StandardScaler()
scaler.fit_transform(X_train)   # learns mean_ and scale_ FROM X_train, then applies
scaler.transform(X_val)         # applies X_train's mean_ and scale_ to X_val
scaler.mean_                    # the learnt state — computed from training rows only

# The bug this prevents, written out by hand:
# X_val_bad = StandardScaler().fit_transform(X_val)   # val rows standardised by THEIR OWN mean

fit_transform(X_val) instead of transform(X_val). It runs, it returns a nicely scaled array, and it quietly gives every validation row information about the other validation rows. The score improves, so it looks like an improvement rather than a mistake.

ColumnTransformer

What it is

ColumnTransformer([(name, transformer, columns), ...]) applies a different transformer to a different subset of columns and concatenates the results side by side. remainder='drop' (the default) discards untouched columns; remainder='passthrough' keeps them.

Why it matters

Real tables are heterogeneous: numeric columns want imputation and scaling, categorical columns want one-hot encoding, and free text wants a vectoriser. Without a ColumnTransformer these three paths are three manual slices that you must reassemble in exactly the same order at serving time.

In the real world: Scaling numbers, encoding categories and processing text in one place for a loan-application model.

Visual example

ColumnTransformer

One table in, parallel branches by column, one matrix out — in a fixed, recorded order.

numeric branch

(impute median -> scale)

('cat', OneHotEncoder(), ['city'])

-> city=Pune, city=Delhi

remainder='drop'

signup_ip never reaches the model

X_train

	age	income	city	signup_ip
0	34	52000	Pune	10.0.0.4
1	41	61000	Delhi	10.0.0.9
2	51	48000	Pune	10.0.1.2

pre.get_feature_names_out() -> ['num__age', 'num__income', 'cat__city_Delhi', 'cat__city_Pune']

Why this matters

The output column order is decided by the order of the transformer list, not by the order of the original dataframe. That ordering is part of the fitted artifact, and it is what makes a raw dataframe at serving time land in the right slots.

Common misconception

Wrong idea: a ColumnTransformer runs its branches in sequence, each one seeing the previous one's output.

Correct idea: the branches run in parallel on disjoint column subsets and their outputs are concatenated. A step that should see the scaled numbers must live *inside* the numeric branch, or *after* the whole ColumnTransformer.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder

num = Pipeline([('impute', SimpleImputer(strategy='median')), ('scale', StandardScaler())])
cat = Pipeline([
    ('impute', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore')),
])

pre = ColumnTransformer([
    ('num', num, ['age', 'income']),
    ('cat', cat, ['city', 'plan']),
])
pipe = Pipeline([('pre', pre), ('clf', LogisticRegression(max_iter=1000))])
pipe.fit(X_train, y_train)
pre.get_feature_names_out()[:4]   # names survive the transformation

remainder='drop' silently deletes every column you forgot to list. A feature added to the dataframe later is simply absent from the model, no error raised — and the reverse mistake, remainder='passthrough', will happily pass an unscaled raw string column into an estimator that cannot take it.

Custom transformers

What it is

A class with fit(self, X, y=None) returning self and transform(self, X) returning the transformed data. Inheriting BaseEstimator gives you get_params/set_params (needed for cloning and grid search); TransformerMixin gives you fit_transform for free.

Why it matters

Domain feature engineering — a ratio, a log, a bucketing rule, a date difference — belongs inside the pipeline, not in a preparatory script. Wrapped as a transformer it is cloned and refit per fold like every other step, and it is serialised along with the model.

In the real world: Packaging the company's own “days since last order” logic so every model uses it the same way.

from sklearn.base import BaseEstimator, TransformerMixin
import numpy as np

class LogSkewed(BaseEstimator, TransformerMixin):
    def __init__(self, threshold=1.0):
        self.threshold = threshold        # store the arg unchanged, under the same name

    def fit(self, X, y=None):
        skew = np.abs(np.mean((X - X.mean(0)) ** 3, axis=0) / (X.std(0) ** 3 + 1e-12))
        self.skewed_ = skew > self.threshold   # learnt state: trailing underscore
        return self                             # fit MUST return self

    def transform(self, X):
        out = np.asarray(X, dtype=float).copy()
        out[:, self.skewed_] = np.log1p(out[:, self.skewed_])
        return out

Pipeline([('log', LogSkewed()), ('scale', StandardScaler()), ('clf', LogisticRegression())])

Two rules break cloning and therefore break GridSearchCV. __init__ must only assign its arguments verbatim to attributes of the same name — no validation, no derived values — and anything learnt must be set in fit with a trailing underscore. A transformer that computes state in __init__ is refit by nobody and leaks the full dataset into every fold.

Pipelines inside cross-validation

What it is

cross_val_score(pipe, X, y, cv=5) clones the *whole* pipeline for each fold and refits every step on that fold's training portion only. The imputer's medians, the scaler's statistics and the encoder's vocabulary are therefore recomputed five times, from five different subsets.

Why it matters

This is where a pipeline earns its keep. On a single train/test split, preprocessing by hand before splitting is a mistake you might get away with; inside k-fold cross-validation the held-out fold is part of the data every manual preprocessing step already saw, so every one of the k scores is optimistic.

In the real world: Keeping data preparation inside each test slice, so the reported churn accuracy is honest.

Visual example

5-fold CV

The same model and the same data: preprocessing fitted once on everything versus refit inside each fold.

Leaky: scaled + selected before CVPipeline refit per foldHeld-out test score

Which estimate told the truth

leaky: mean CV AUC 0.944, spread 0.011 — confident and wrong

pipeline: mean CV AUC 0.796, spread 0.024 — noisier and right

Test AUC 0.79 — within one standard deviation of the pipeline estimate only

Why this matters

Leakage does not just inflate the mean; it also shrinks the spread, because every fold is drawing on the same global statistics. A suspiciously tight set of fold scores is itself a leakage smell.

Common misconception

Wrong idea: scaling before cross-validation is harmless because scaling does not use the labels.

Correct idea: it uses the held-out rows, which is enough. And label-free is not the boundary anyway: a mean computed over data you are about to score is information from the future, whether or not y was involved.

from sklearn.model_selection import cross_val_score

# Correct: preprocessing is refit inside every fold.
scores = cross_val_score(pipe, X, y, cv=5, scoring='roc_auc')
print(scores.mean(), scores.std())

# Leaky: one scaler fitted on ALL rows, then cross-validated.
X_all = StandardScaler().fit_transform(X)
leaky = cross_val_score(LogisticRegression(max_iter=1000), X_all, y, cv=5, scoring='roc_auc')
print(leaky.mean())     # reliably higher, and reliably wrong

Feature *selection* is the version of this that bites hardest. Choosing the top-k features by correlation with y over the whole dataset and then cross-validating the selected features can turn pure noise into an apparently strong model, because the selection step consumed the labels of every fold.

Practise it: A Pipeline That Does Not Leak, A Split That Does Not Leak Across Groups

Tuning a pipeline with nested parameter names

What it is

In a grid or random search, a pipeline's parameters are addressed with a double underscore path: step__param, and outer__inner__param for nested pipelines. GridSearchCV(pipe, {'pre__num__impute__strategy': ['mean', 'median'], 'clf__C': [0.1, 1, 10]}) searches preprocessing and model choices together.

Why it matters

Preprocessing decisions are hyperparameters and interact with the model's. Whether to impute with the mean or the median, whether to log-transform, how many components to keep — each of these is only 'best' relative to a particular regulariser strength, so tuning them jointly is the only honest way.

In the real world: Tuning both the preparation steps and the model's settings in one search.

from sklearn.model_selection import GridSearchCV

grid = {
    'pre__num__impute__strategy': ['mean', 'median'],
    'pre__num__scale__with_mean': [True, False],
    'clf__C': [0.1, 1.0, 10.0],
}
search = GridSearchCV(pipe, grid, cv=5, scoring='roc_auc', n_jobs=-1)
search.fit(X_train, y_train)
search.best_params_
search.best_estimator_          # a pipeline, refit on all of X_train

You can also swap a whole step: {'clf': [LogisticRegression(), RandomForestClassifier()]} is a legal grid entry. What is not legal is a single underscore — clf_C — which raises a long ValueError listing every valid key, and is almost always just a typo for clf__C.

Practise it: A Pipeline That Does Not Leak

Persisting the whole pipeline

What it is

joblib.dump(pipe, 'model.joblib') writes the fitted pipeline — transformers, learnt statistics, category vocabulary and estimator — as one file. joblib.load returns something whose predict accepts the same raw dataframe columns that fit was given.

Why it matters

Saving only the estimator is the classic cause of training-serving skew. The serving code then re-implements the preprocessing, slightly differently, and the model receives inputs from a distribution it was never trained on while every test passes.

In the real world: Saving preparation and model together, so the service scoring live payments can't forget a step.

Visual example

Training-serving skew

Ship the estimator alone and the serving side must re-derive every learnt constant, by hand.

model.joblib holding only clf

needed at serving	where it lives
coefficients	in the file
imputation medians	a notebook cell
scaler mean / sd	a notebook cell
category vocabulary	nowhere

what gets shipped, and what the serving code has to invent

model.joblib holding the pipeline

needed at serving	where it lives
coefficients	in the file
imputation medians	in the file
scaler mean / sd	in the file
category vocabulary	in the file

Why this matters

Every row in the left table is a number a second engineer will eventually guess. Guessed constants are indistinguishable from correct ones in a unit test and produce a quietly worse model in production.

Common misconception

Wrong idea: the model is the weights.

Correct idea: the model is the weights plus every constant that decides what the weights are applied to. A median, a standard deviation and a category list are as much part of the artifact as a coefficient.

import joblib, sklearn, pandas as pd

joblib.dump({'pipeline': pipe, 'sklearn_version': sklearn.__version__,
             'features': list(X_train.columns), 'trained_at': '2026-09-19'}, 'model.joblib')

bundle = joblib.load('model.joblib')
row = pd.DataFrame([{'age': 34, 'income': 52000, 'city': 'Pune', 'plan': 'pro'}])
bundle['pipeline'].predict_proba(row[bundle['features']])[:, 1]

A pickled pipeline is not a portable format: it stores references to the classes that made it, so loading it under a different scikit-learn version may warn, may silently behave differently, or may fail outright. Pin the library version next to the file, and test that loading it and scoring a known row reproduces a known output.

Practise it: A Pipeline That Does Not Leak

Reading a pipeline as documentation

What it is

A pipeline's repr is a complete, executable statement of what happens to the data. pipe.get_params() lists every knob, pipe[:-1].transform(X) yields the matrix the estimator actually receives, and set_output(transform='pandas') makes the intermediate frames come back with column names attached.

Why it matters

Inheriting a project, the pipeline definition is the one artifact that cannot be out of date: the preprocessing it describes is the preprocessing that ran. A README describing the preprocessing is a claim; a pipeline is the implementation.

In the real world: A new engineer can see exactly how a fraud model's inputs are prepared, just by reading its pipeline.

pipe.set_output(transform='pandas')        # intermediate steps keep column names
X_model = pipe[:-1].transform(X_train)      # exactly what the estimator is fed
X_model.columns[:5]
X_model.shape                               # 14 raw columns became 37 features

list(pipe.named_steps)                      # ['pre', 'clf']
pipe.named_steps['pre'].get_feature_names_out()[:3]

pipe[:-1] slices a *fitted* pipeline without refitting it, which is what you want for inspection — but pipe[1:] is not a model you can fit on raw data, because the first step's transformation is missing. Slices are for looking, not for training.

Practise it: A Pipeline That Does Not Leak

The feature engineered outside the pipeline

What it is

The most common remaining leak: a column computed on the full dataframe — a target-encoded category, a group mean, a z-score, a 'days since last purchase' built from a timestamp later than prediction time — added before the split and then carried innocently through a perfectly correct pipeline.

Why it matters

The pipeline's guarantee only covers what is inside it. A feature manufactured upstream of the split has already mixed train and test rows, and no amount of correct fit/transform discipline downstream can unmix them.

In the real world: A feature computed in a notebook but not in production quietly breaks the live model.

Visual example

Leakage signature

A group-mean feature computed before the split: validation soars, production does not move.

Validation AUCLive production AUC

Reading the gap

features 1-3: val and prod move together — the gains are real

feature 4: val +0.15, prod -0.01 — the gain exists only where test rows were visible

Rule of thumb: a gain that does not appear offline AND online is not a gain

Why this matters

Leakage is easiest to catch as a *divergence* between two curves that had been tracking each other. One curve alone, however carefully validated, cannot show you this.

Common misconception

Wrong idea: if the pipeline is correct, the experiment is leak-free.

Correct idea: the pipeline only guards what it contains. Every column that existed before train_test_split was called is your responsibility to justify, one at a time.

# Leaky: the mean is taken over every row, test rows included.
df['city_mean_spend'] = df.groupby('city')['spend'].transform('mean')
X_train, X_test = split(df)

# Correct: the encoding is a fitted step, refit per fold inside the pipeline.
from sklearn.preprocessing import TargetEncoder
pre = ColumnTransformer([
    ('cat', TargetEncoder(target_type='continuous'), ['city']),
    ('num', StandardScaler(), ['age', 'income']),
])
pipe = Pipeline([('pre', pre), ('model', Ridge())])

The symptom is a validation score that jumps when a feature is added and then fails to reproduce in production. Before trusting any gain, ask one question of the new feature: could it have been computed using only the single row being predicted plus state fitted on the past? If not, it belongs inside the pipeline or nowhere.

Practise it: A Pipeline That Does Not Leak