ML System Design · Module 7 of 47 · 11 concepts · about 14 min

Sampling, Imbalance and Bias

How the examples a model learns from can mislead it: rare classes, sampled negatives, splits that leak the future, users who appear on both sides of a split, and the biases created by what the product chose to show.

At work: Where most 'it worked offline but not in production' stories begin. A model is only as honest as the sample it was trained and tested on.

Training data is a sample

What it is

A model learns from a sample of the world, and is used on the whole of it. The sample must look like the cases the model will meet:

the same kinds of users, products and situations;
in roughly the same proportions, or with the difference corrected for;
from a recent enough period that behaviour has not changed.

Why it matters

Whatever the sample leaves out, the model does not know.

A delivery model trained only on weekday orders has never seen a festival rush.
A recommender trained only on loyal customers does badly for new ones.
The gap between the sample and the real traffic is one of the main reasons offline scores do not survive launch.

In the real world: Kirana Kart samples training orders from every city and every hour, and checks that each city's share of the training set roughly matches its share of live orders.

Training on whatever is easiest to export. 'Orders from our biggest city last month' is convenient, and teaches a model that every city looks like that one.

Class imbalance

What it is

A problem is imbalanced when one class is much rarer than the other:

fraud: well under 1% of payments;
machine failures: a tiny fraction of hours;
clicks on ads: a few in a thousand.

The rare class is almost always the one that matters.

Why it matters

Imbalance breaks naive training and naive measurement.

A model can score high accuracy by always predicting the common class, while catching nothing.
Training sees so few positives that it may not learn their pattern.
Measures that focus on the rare class (precision, recall, PR-AUC, module 16) are needed instead.

In the real world: Nearly every fraud, spam, failure-prediction and ad-click system is heavily imbalanced; interviewers check that candidates notice immediately.

Visual example

Computed

100,000 payments, 0.5% fraud. A model that never says 'fraud' is 99.5% accurate and catches none.

The 'always legitimate' model

	really fraud	really legitimate
predicted fraud	0	0
predicted legitimate	500	99,500

Scores

acc: accuracy = 99,500 / 100,000 = 99.5%

rec: fraud caught (recall) = 0 / 500 = 0%

Reporting accuracy for an imbalanced problem. A fraud model that never flags anything is 99.5% accurate and completely useless.

Practise it: Design a Real-Time Fraud Detection System

Handling class imbalance

What it is

Several remedies, usually combined:

Use the right measures: precision, recall and PR-AUC, not accuracy.
Class weights: make each mistake on the rare class count more in the loss.
Resampling: keep fewer common examples (undersampling) or repeat rare ones (oversampling).
Tune the threshold for the business cost of each mistake, instead of using 0.5.
Get more positives: label more rare cases, often with active learning (module 6).

Why it matters

Modern models (especially boosted trees) handle imbalance better than people expect.

Often the biggest gain is simply measuring correctly and choosing the threshold well.
Resampling changes the predicted probabilities, which matters if anything downstream uses them as real probabilities.

In the real world: Kirana Kart's fraud model uses class weights and keeps all fraud cases, keeps 10% of legitimate payments, and corrects its probabilities afterwards so the risk team can read them as real chances.

Oversampling before splitting into training and test. Copies of the same fraud case then appear on both sides, and the test score is inflated.

Practise it: Design a Real-Time Fraud Detection System

Negative sampling and correcting probabilities

What it is

When negatives vastly outnumber positives (items shown but not chosen), training keeps only a share of them. This shifts the model's scores upward, because it saw positives more often than they really happen. If a share w of negatives was kept:

the model's odds are too high by a factor of 1/w;
to correct: p = q·w / (q·w + 1 − q), where q is the model's score.

Why it matters

Whether you need the correction depends on how the score is used.

For ranking only the order matters, and sampling does not change the order, so no correction is needed.
For anything that uses the number itself (expected revenue, ad pricing, a threshold set in probability terms), uncorrected scores are badly wrong.

In the real world: Facebook's well-known ad click paper described exactly this re-calibration after down-sampling negatives, because ad prices depend on the predicted probability itself.

Visual example

Computed

Negatives kept at 10%. The model's raw scores, and the real probabilities after correction.

p = q·w / (q·w + 1 − q), with w = 0.1

model score q	corrected probability p
0.9	0.4737
0.5	0.0909
0.2	0.0244
0.05	0.0052

The order is unchanged, so a ranking would be the same either way.

def correct_for_negative_sampling(q: float, keep_rate: float) -> float:
    """Undo the upward shift from keeping only keep_rate of negatives."""
    return q * keep_rate / (q * keep_rate + (1 - q))

correct_for_negative_sampling(0.5, 0.1)   # 0.0909...

Feeding sampled-model scores into an ad auction or a cost calculation. Every predicted click rate is several times too high, and so is every bid.

Practise it: Design a Recommendation System

Split by time

What it is

For anything that changes over time, split data by date:

train on an earlier period;
validate on the period after it;
test on the period after that.

For a more thorough check, repeat this over several cut-off dates (rolling or backtesting evaluation).

Why it matters

In production, the model always predicts the future from the past.

A random split lets the model learn from days that come after some test days, which never happens live.
It hides drift: a time split shows how fast accuracy falls as the test period gets further from training.

In the real world: Kirana Kart trains its delivery-time model on twelve months up to two weeks ago, validates on the week after, and tests on the most recent week, mirroring how it will be used.

Visual example

Time split

The model never sees anything later than the period it is judged on.

Training, validation and test, in date order

TrainMonths 1 to 12
GapDays for labels to settle
ValidateNext week: tune here
TestWeek after: score once

A random split on time-ordered data. Offline scores look excellent because tomorrow is partly in the training set, and real performance is much lower.

Split by group

What it is

When many rows belong to one entity (a user, a store, a device), keep all of an entity's rows on the same side of the split:

a user's orders are all in training, or all in test;
a store's days are all in training, or all in test, if the model must work for new stores.

Why it matters

Otherwise the model can memorise the entity instead of learning the pattern.

If a user's other orders are in training, the model 'recognises' them in the test set.
The score then measures memory of known users, and drops sharply for new ones.
Which grouping to use depends on what the model must generalise to.

In the real world: A medical-imaging team discovers its high score came from several scans of the same patient landing on both sides of a random split; grouping by patient cuts the score sharply.

Testing a new-store model on new days from known stores. It will look fine and fail on the first store it has never seen.

Selection bias

What it is

Selection bias is when the examples you have are not chosen at random from the cases the model will see:

only users who left a rating;
only loans that were approved;
only items that were shown;
only customers who answered a survey.

Why it matters

The model learns the selected group, not the population.

People who rate are more extreme than people who do not.
Approved loans look safer than all applicants.
Corrections include random holdouts, importance weighting (giving under-represented cases more weight), and modelling the selection itself.

In the real world: A food app's dish-rating model learns that almost every dish is either excellent or terrible, because those are the only orders people bother to rate.

Assuming more data fixes selection bias. Ten times as many ratings from the same kind of rater is still the same bias.

Practise it: Diagnose Training-Serving Skew

Position bias

What it is

Items higher on the screen are clicked more just because they are higher:

people look at the top first and often stop there;
so a click partly measures position, not only how good the item is.

To separate the two, estimate how likely each position is to be looked at (its propensity), for example from a small experiment that shuffles positions at random, then divide it out.

Why it matters

Uncorrected, position bias feeds itself.

The model learns that items it placed at the top are good, so it keeps placing them at the top.
New items, shown lower, never get a fair chance.
Corrections: inverse propensity weighting in training, position as a feature during training (set to a fixed value at serving), and small randomised experiments.

In the real world: Search engines and e-commerce sites run small, controlled position swaps to estimate how much attention each slot gets, and use the result to correct their training data.

Visual example

Computed

Five slots on a results page. After dividing out how often each slot is looked at, all five items are equally good.

Illustrative. Chance of being looked at comes from a randomised test.

position	observed click rate	chance slot is looked at	click rate if looked at
1	10%	100%	10%
2	7%	70%	10%
3	5%	50%	10%
5	3%	30%	10%
10	1.5%	15%	10%

Every difference in the observed column was caused by position alone.

Training on raw clicks as if every position had the same chance to be seen. The model will rank yesterday's top items first for ever.

Practise it: Design a Recommendation System, Design a Search Ranking System

Stratified sampling for evaluation

What it is

An overall score can hide groups the model fails. Stratify evaluation:

split the test set into meaningful groups (slices): city, new versus returning user, device, product category;
make sure each slice has enough examples to measure;
report the score per slice, not only overall.

Why it matters

Averages hide failures.

A delivery model that is excellent in big cities and poor in small towns can have a fine average.
Small but important slices (new users, high-value orders) may barely move the overall number.
Interviewers like hearing 'and I would check the slices'.

In the real world: Kirana Kart reports its delivery-time error for each city, for new stores, and for rainy days separately; a release that improves the average but worsens rainy days is held back.

Drawing a random test set too small to measure rare slices. With 50 orders from a small town, the town's score is noise.

Practise it: Design Model Monitoring

Interview questions: sampling, imbalance and bias

What it is

The questions interviewers ask about sampling, imbalance and bias, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: A common follow-up in recommendation interviews is 'your training data only contains items you showed; what does that do?'. It is testing exactly this module.

Q: How do you handle class imbalance?

Measure with precision, recall and PR-AUC rather than accuracy; use class weights or resampling; tune the threshold for the cost of each mistake; and get more positive labels where possible.

Q: Why correct probabilities after negative sampling?

Keeping a fraction w of negatives inflates the odds by 1/w. Ranking is unaffected, but anything using the probability itself, like bidding or expected value, needs p = qw / (qw + 1 − q).

Q: Why split by time?

Production always predicts the future from the past. A random split leaks later information into training and hides drift, so offline scores are too optimistic.

Q: What is position bias and how do you correct it?

Higher items get more clicks regardless of quality. Estimate each position's chance of being seen, for example with randomised swaps, then weight training examples by the inverse, or use position as a training-only feature.

Q: What is selection bias? Give an example.

Training data chosen non-randomly, like repayment outcomes only for approved loans. The model learns the selected group; random holdouts and importance weighting help.

Q: Why evaluate on slices?

Averages hide groups where the model fails, such as new users or small cities. Per-slice scores catch regressions that the overall number misses.

Saying 'I'd use SMOTE' as the whole answer to imbalance. Measuring and thresholding correctly matter more.

Practise it: Design a Recommendation System, Design Model Monitoring, Diagnose Training-Serving Skew, Design a Personalized News Feed, Design a Real-Time Fraud Detection System, Design a Search Ranking System

Exercise: watch a leaky split lie

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: Reviewers of ML work routinely ask 'how did you split?' before reading any results; this exercise is why.

Your task. Show, on real data, how much a random split and an ungrouped split overstate a model's accuracy.

Steps

Use a public time-stamped dataset with repeated entities, for example retail sales by store and day, or taxi trips with timestamps.
Train the same model three ways: random split; time split (train earlier, test later); time split grouped by entity (test entities never seen in training).
Record the error for each.
Add a column showing the test period's distance in weeks from the training cut-off, and plot error against it.
For a classification target, compute accuracy and recall for a model that always predicts the majority class.

You are done when

You have three scores and can explain why they are in the order they are.
Your plot shows whether accuracy falls as the test period moves further from training.
You can state the majority-class accuracy and why it is meaningless.

Stretch. Simulate position bias: generate clicks where the chance of looking falls with position, train on raw clicks, then with inverse propensity weights, and compare which recovers the true item quality.