Machine Learning · Module 2 of 26 · 9 concepts · about 16 min

The ML Problem-Solving Framework

Turning a vague business question into a well-posed prediction problem with a target, a unit, a baseline, and a metric someone will act on.

At work: Turning a manager's question (“why are customers leaving?”) into a model worth building, and knowing when not to build one.

Framing a business problem as an ML problem

What it is

A usable framing states four things: the decision the prediction will inform, the unit that is predicted about, the target variable, and the moment of prediction. If any one is missing, the problem is not yet specified.

Why it matters

Most failed ML projects are well-executed answers to unstated questions. 'Reduce churn' does not say whether you need a churn probability per user per month, a ranked list of at-risk accounts for ten sales calls, or an estimate of a discount's effect — and those are three different models with three different datasets.

In the real world: “Reduce churn” becomes: predict which subscribers are likely to cancel in the next 30 days, so the retention team can call them first.

Visual example

Problem framing

Four slots. A framing with a blank in any of them cannot be evaluated, because nobody knows what right looks like.

Fix this first

(everything else follows from the decision)

"Reduce churn", made specific

	vague version	specified version
Decision	"act on churn risk"	call 500 accounts this week
Unit	"customers"	one account, one week
Target	"churn"	cancels within 30 days (0/1)
Moment	"soon"	Monday 06:00, data to Sunday 23:59

target + moment

together they define one training row

The right-hand column is a dataset specification

Why this matters

The specified column can be turned into SQL and a label; the vague column cannot. Writing it out is the cheapest step in the whole project and the one most often skipped.

Common misconception

Wrong idea: framing is paperwork you can backfill once the model works.

Correct idea: the framing is the dataset. Change the moment of prediction and every feature has to be recomputed, so a late change usually means starting over.

# A framing, written down before any code, as a dictionary you can review.
framing = {
    "decision": "which 500 accounts the retention team calls this week",
    "unit": "one account, once per week",
    "target": "cancels within the next 30 days (0/1)",
    "prediction_time": "Monday 06:00, using data available up to Sunday 23:59",
    "metric": "precision@500, plus retained revenue as the business check",
}

Letting the available data define the target. 'We have a cancellation flag, so let us predict cancellation' skips the question of whether anyone can act on that prediction — and an unactionable prediction has no value however accurate it is.

Choosing the task type: classification, regression, ranking, clustering

What it is

The task type follows from the shape of the output the decision needs: a category (classification), a number (regression), an order over candidates (ranking), or groups with no ground truth at all (clustering).

Why it matters

Task type determines the model family, the loss, and the metric in one step. Picking it by habit — classification because you have seen classification — produces models that answer a question adjacent to the one asked.

In the real world: “Will they buy?” is classification, “how much will they spend?” is regression, and “which products to show first?” is ranking.

Visual example

Task types

The output shape the decision needs picks the task — not the data you happen to have.

Often mis-framed

(ranking is scored at a cutoff, not overall)

Four task types

	output	example question	typical metric
Classification	a class or probability	will this user cancel?	PR-AUC, recall@k
Regression	a real number	how many units next week?	RMSE, MAE
Ranking	an ordering	which 20 items to show?	precision@k, NDCG
Clustering	groups, no labels	what segments exist?	silhouette, human review

Pick the row whose output the decision consumes

Why this matters

Ranking is the common case people frame as classification: the team can only call 500 accounts, so overall accuracy is irrelevant and precision in the top 500 is everything.

Common misconception

Wrong idea: a probability output means the task is classification.

Correct idea: a probability is just the score. If the decision is 'which top-k do we act on', the task is ranking, and it must be measured at that k.

from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.cluster import KMeans

LinearRegression()      # regression: how much will this order be worth?
LogisticRegression()    # classification: will this order be refunded?
KMeans(n_clusters=5)    # clustering: what kinds of orders exist at all?
# Ranking: score with a regressor/classifier, then sort — and evaluate at k.

Bucketing a continuous target into classes to make it 'easier'. Predicting spend as low/medium/high throws away the information in the number, introduces arbitrary boundaries, and makes a £99 error at a £100 threshold count the same as a £9,000 one.

Practise it: k-Means, and the Cluster That Loses All Its Points

Defining the target variable

What it is

The target is a rule, written in code, that turns raw history into a 0/1 or a number for each row — including the window it looks over and the exact cutoff. 'Churn' becomes, say, 'no session in the 30 days following the prediction date'.

Why it matters

Two reasonable definitions of the same word produce datasets with different positive rates, different difficulty, and non-comparable metrics. The definition also decides how long you must wait before a row can be labelled at all.

In the real world: Deciding that “churned” means no purchase in 60 days, not 30, changes who the model flags and which customers get offers.

Visual example

Label definition

The label is computed from a window strictly after the prediction date; features come only from before it.

30-day label window, after the cutoff

input

w-4

w-3

w-2

w-1

cut

w+1

w+2

w+3

w+4

output

NaN

feature side

label side

churn = 1

Why this matters

Drawing the timeline is how leakage gets caught before it is coded. If a feature's window crosses the cutoff, it cannot exist at prediction time in production.

Common misconception

Wrong idea: as long as the target is in a different column, the data is fine.

Correct idea: what matters is when each value became known. A column that only fills in after the event is a label in disguise, whatever it is named.

1Pick a cutoffThe moment the prediction would actually be made.
2Features: beforeOnly data recorded at or before the cutoff.
3Label: afterThe outcome over a fixed horizon following it.

Anything computed from w+1 onward is a label; anything computed from w-1 back is a feature. Mixing the two is the leakage that inflates every offline score.

import pandas as pd

# Label window: strictly after the prediction date, so no future info leaks in.
cutoff = pd.Timestamp("2026-03-01")
horizon = cutoff + pd.Timedelta(days=30)

future = sessions[(sessions["ts"] > cutoff) & (sessions["ts"] <= horizon)]
active_later = set(future["user_id"])
labels = users.assign(churn=(~users["user_id"].isin(active_later)).astype(int))
labels["churn"].mean()      # the positive rate your metrics will live with

Defining the target with a window that includes the prediction moment itself, or any signal that only exists because the event happened. A 'cancellation reason' field is populated only for cancelled accounts — include it and the model scores 0.99 offline and is worthless live.

The unit of prediction

What it is

The unit is what one row of X represents — one user, one user-week, one session, one transaction. It fixes the row count, the aggregation level of every feature, and what a single prediction is about.

Why it matters

Choosing it wrongly makes the metric misleading rather than wrong. Per-session accuracy of 0.9 can coexist with getting every heavy user's outcome wrong, because heavy users contribute hundreds of easy rows each.

In the real world: Predicting churn per customer per week, not per customer once, matches how often the retention team actually acts.

Visual example

Unit of prediction

One user can be one row or fifty. The unit decides whose outcome a metric is really averaging over.

Same people

(different row counts)

Per-user vs per-user-week

	rows if unit = user	rows if unit = user-week
heavy user	1	52
light user	1	3
metric effect	each user counts once	heavy users dominate

Group-aware splitting is mandatory once a unit repeats an entity

Where the 0.9 comes from

per-user-week: 52 easy weeks x 1 heavy user = 52 correct rows

3 weeks x 1 light user, all wrong = 3 incorrect rows

accuracy 52/55 = 0.95, while one of two users is fully mispredicted

Why this matters

It explains a class of 'great offline, useless online' results: the metric was averaging rows when the business cares about entities.

Common misconception

Wrong idea: more rows is always better, so the finer unit is the better one.

Correct idea: the finer unit adds correlated rows, not independent ones. It inflates the apparent sample size and hands the model an identity feature to exploit.

# Same events, two units, two very different datasets.
per_user = events.groupby("user_id").agg(n_events=("id", "size"),
                                          last_seen=("ts", "max"))
per_user_week = events.groupby(["user_id", pd.Grouper(key="ts", freq="W")]).size()

per_user.shape, per_user_week.shape      # rows differ by an order of magnitude

Mixing units by accident, then splitting randomly. With one row per user-week, a random split puts the same user in train and test; the model can recognise the user rather than the behaviour. Split by group (GroupKFold on user_id), not by row.

Baselines: beat something dumb first

What it is

A baseline is a trivially simple predictor — the majority class, the mean, last week's value, or the existing hand-written rule — fitted and scored in the same harness as the real model.

Why it matters

It converts an unreadable number into a comparison. An RMSE of 12.4 means nothing; an RMSE of 12.4 against a mean-predictor's 12.6 means the model has learned almost nothing, and that is the single most useful thing to discover early.

In the real world: If “predict last week's sales” is only 5% worse than a complex model, the complex model may not be worth running.

Visual example

Baselining

A score only means something next to a baseline. Here the fancy model has added 0.2 points of nothing.

0.97 accuracy, zero skill

(predicts 'no' every time)

The column that moves

(recall on the rare class)

Same validation split, four predictors

	accuracy	recall on positives
Majority class	0.97	0
Existing hand rule	0.961	0.31
Logistic regression	0.972	0.34
Gradient boosting	0.974	0.36

Report the lift over baseline, never the raw score alone

The arithmetic behind 0.97

3,000 negatives, 100 positives -> 3,100 rows

predict 'no' for all: 3,000 correct / 3,100 = 0.968

gradient boosting: 0.974 -> +0.006 accuracy, +0.36 recall

Why this matters

It is the cheapest possible experiment and the one that most often kills a bad framing before weeks are spent on it.

Common misconception

Wrong idea: baselines are a formality for the write-up.

Correct idea: the baseline is the null hypothesis. Without it you cannot tell a learned signal from the base rate, and the base rate is usually the whole score.

from sklearn.dummy import DummyClassifier, DummyRegressor

DummyClassifier(strategy="most_frequent").fit(X_train, y_train).score(X_val, y_val)
DummyClassifier(strategy="stratified", random_state=0).fit(X_train, y_train)
DummyRegressor(strategy="mean").fit(X_train, y_train).score(X_val, y_val)   # R^2 ~ 0

# Also baseline the incumbent: the current rule, scored on the same split.

Skipping the baseline because the model's score 'looks good'. On a dataset with 97% negatives, 0.97 accuracy is what predicting 'no' always gets you — and teams have shipped exactly that, repeatedly.

The ML project lifecycle

What it is

Problem framing, data collection and labelling, exploration, feature and split design, modelling, evaluation, deployment, monitoring — and back to framing when monitoring shows drift. It is a loop, and the loop runs many times.

Why it matters

Effort is usually spent as if modelling were the bulk of it. In practice data and evaluation dominate, and the failures that hurt most in production come from the stages after modelling, which are the ones beginners never rehearse.

In the real world: A recommendation model goes from question to data, prototype, A/B test, launch and monitoring, and most of the time is spent outside the model.

Visual example

Lifecycle

Modelling is one stage of eight, and it is not where the time or the failures are.

Where the effort and the failures live

	share of effort	classic failure
Framing	5%	unactionable prediction
Data & labels	45%	leakage, wrong label window
Modelling	15%	tuned against the test set
Evaluation	15%	metric nobody acts on
Deploy & monitor	20%	training/serving skew, silent drift

Why this matters

Planning by this table prevents the standard shape of a doomed project — four weeks of model tuning on a dataset whose label was defined wrongly in week one.

Common misconception

Wrong idea: the project is finished when the model hits the target metric.

Correct idea: that is roughly the halfway point. A model that is not deployed, monitored and retrained is a result, not a system.

1Frame and labelDecision, unit, target, moment — then build the dataset from them.
2Model and evaluateBaseline first, then complexity, judged on validation data.
3Deploy and watchShip the whole pipeline, log inputs and outputs, alert on drift.
4LoopWhat monitoring reveals becomes the next iteration's framing.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
import joblib

pipe = Pipeline([("scale", StandardScaler()), ("clf", LogisticRegression(max_iter=1000))])
pipe.fit(X_train, y_train)
joblib.dump(pipe, "model.joblib")    # the whole preprocessing + model artifact

# Monitoring: log inputs and predictions, and recompute the metric when labels arrive.

Shipping the model without the preprocessing. If the scaler lives in a notebook cell and only the estimator is serialised, production feeds raw values into a model fitted on standardised ones — no error, badly wrong predictions. Serialise the whole Pipeline.

Proxy metrics versus business metrics

What it is

The business metric is the outcome you are paid for — retained revenue, resolved tickets, conversion. The proxy is what you can compute offline from a static dataset — AUC, RMSE, precision@k. They correlate only over a range, and the relationship is not monotonic.

Why it matters

Optimising a proxy past the point where it tracks the business metric produces the classic result: offline scores keep improving and nothing changes in production, because the remaining gains are in regions the decision never touches.

In the real world: A model that raises click-through but lowers actual purchases is a win on the chart and a loss for the business.

Visual example

Metric divergence

The proxy keeps climbing after the business metric stops. Past the elbow, offline gains buy nothing.

Offline AUC (proxy)Retained revenue (business)

Why this matters

It tells you when to stop tuning. The signal to stop is not a plateau in the offline metric — it is the divergence, which only a business-side measurement can reveal.

Common misconception

Wrong idea: a better offline metric is always a better model.

Correct idea: better on the proxy. Beyond the divergence point the proxy is measuring rearrangements of cases the decision never reaches.

Versions 5-8 improve the proxy by 8 points and the business metric by nothing. The work was real; the value was not.

# Score the proxy on the same rows the decision will actually cover.
import numpy as np

order = np.argsort(-proba_val)[:500]        # the 500 accounts the team can call
precision_at_500 = y_val[order].mean()
value_at_500 = revenue_val[order][y_val[order] == 1].sum()
precision_at_500, value_at_500              # proxy, and the thing it stands in for

Reporting a global metric for a top-k decision. AUC over all 300,000 accounts improves by rearranging the bottom of the list, which nobody will ever call — and it can improve while precision in the top 500 falls.

Real-world constraints: latency, interpretability, cost, fairness

What it is

Non-accuracy requirements that narrow the model menu before accuracy is considered: a 20 ms budget, a regulator who needs a reason per decision, a retraining bill, an obligation not to differ in error rate across groups.

Why it matters

Each constraint eliminates families outright. A model that cannot be explained cannot be deployed in credit decisioning however accurate it is, and 'we will add explanations later' is not usually possible.

In the real world: A payment fraud check must answer in under 100 ms, and a loan model must be explainable to regulators; both limit the model choice.

Visual example

Model selection

Constraints are filters applied before accuracy, and they usually decide the model family on their own.

Often the only legal option

(regulated decisions)

Same problem, three candidate families

	latency	explainable per decision	retrain cost
Logistic regression	<1 ms	yes — signed coefficients	minutes
Gradient boosting	2-10 ms	partly — SHAP, approximate	tens of minutes
Deep network	10-50 ms	hard	hours, GPU

Accuracy is the tiebreak among the rows that survive

Why this matters

It reorders the workflow: establish the constraints, filter the families, and only then compare scores. Done the other way round, the best model is often the one you cannot ship.

Common misconception

Wrong idea: pick the most accurate model, then engineer around the constraints.

Correct idea: constraints are hard filters. You cannot retrofit a 20 ms budget or a per-decision reason onto a family that does not offer them.

import time
import numpy as np

start = time.perf_counter()
_ = model.predict(X_val[:1000])
latency_ms = (time.perf_counter() - start) / 1000 * 1000

# Fairness: the same metric, computed per group, then compared.
for g in np.unique(groups_val):
    m = groups_val == g
    print(g, model.score(X_val[m], y_val[m]))

Measuring latency in a notebook on a batch of 1,000 rows and assuming single-row serving is 1/1000th of it. Per-call overhead dominates at batch size one, and a model that scores 1,000 rows in 40 ms can take 15 ms for one.

Start simple, add complexity only when it is justified

What it is

The discipline is to ship the simplest thing that beats the baseline, measure it, and let each added layer of complexity prove its keep on validation data — with the cost of maintaining it counted.

Why it matters

Simple models fail loudly and are debuggable; complex ones fail quietly. And the usual finding is that better features, a better label and more data beat a better algorithm on the same data, so complexity spent on the model is often complexity spent in the wrong place.

In the real world: A simple logistic regression for lead scoring often ships in a week and does 90% as well as a deep model that takes months.

from sklearn.model_selection import cross_val_score
import numpy as np

for name, est in candidates.items():      # baseline first, then upward
    scores = cross_val_score(est, X_train, y_train, cv=5, scoring="average_precision")
    print(f"{name}: {scores.mean():.3f} +/- {scores.std():.3f}")

# A gain smaller than the fold-to-fold spread is not a gain.

Reading a 0.004 improvement as real when the cross-validation standard deviation is 0.02. That difference is noise, and adopting it buys a more complicated system for nothing. Compare the gain against the spread before believing it.