ML System Design · Module 17 of 47 · 13 concepts · about 16 min

Online Experimentation and A/B Testing

Proving that a model helps real users: how A/B tests work, how many users and days they need, choosing what to randomise, guardrails, the traps of peeking, novelty and broken splits, faster methods such as variance reduction and interleaving, and when bandits fit better.

At work: The final word on whether a model ships at almost every consumer tech company. Thousands of experiments a year run at the largest ones.

Why test online

What it is

An online experiment shows the new model to some real users and the old one to others, and compares what they do. It is the only way to measure:

how users react to the new predictions;
the effect on product and business metrics, not proxies;
side effects on guardrails such as speed and complaints.

Why it matters

Offline wins often fail to carry over (module 16).

The experiment measures cause and effect: random assignment means the only systematic difference between the groups is the model.
It replaces opinion ('the new recommendations look better') with evidence.

In the real world: Microsoft, Google, Amazon and Booking.com have each reported that most ideas tested do not improve the target metric, which is exactly why they test everything.

Launching to everyone and comparing this week with last week. Weather, holidays, marketing and news all change between weeks; a before-and-after comparison cannot separate the model's effect from everything else.

How an A/B test works

What it is

The parts of an A/B test:

a hypothesis: 'the new ranker increases basket additions from recommendations';
a randomisation unit, usually the user, assigned to control (old model) or treatment (new model) by a hash of their ID;
a primary metric decided in advance, plus guardrail metrics;
a sample size and duration decided in advance;
a statistical test at the end: is the difference larger than chance would produce?

Why it matters

Fixing these in advance is what makes the result trustworthy.

Choosing the metric after seeing the data invites finding whichever one happened to move.
A hash of the user ID keeps each user in the same group on every visit.

In the real world: Kirana Kart's experiment service assigns users by hashing the user ID with the experiment's name, so different experiments split users independently.

Visual example

A/B test

Everything above the line is decided before the test starts.

From hypothesis to decision

Decided before the test
HypothesisWhat should change, measured how
Size itUsers needed; at least a week
RandomiseHash of user ID
During and after
Run and checkSample ratio, guardrails, no peeking
AnalyseDifference, interval, slices
DecideShip, iterate or stop

import hashlib

def assign(user_id: str, experiment: str, treatment_share: float = 0.5) -> str:
    """Same user, same experiment -> same group, on every request and every server."""
    h = int(hashlib.sha256(f"{experiment}:{user_id}".encode()).hexdigest(), 16)
    return "treatment" if (h % 10_000) < treatment_share * 10_000 else "control"

Randomising per request for a user-facing feature. The same person sees the old model on one screen and the new one on the next, and neither group's behaviour reflects either model.

Sample size and power

What it is

Before running, compute how many users each group needs to detect the smallest effect worth shipping:

baseline rate of the metric;
minimum detectable effect: the smallest change you care about;
significance (usually 5%): the accepted chance of a false win;
power (usually 80%): the chance of detecting a real effect of that size.

For a rate, users per group ≈ 2 × (1.96 + 0.84)² × p(1 − p) ÷ (effect)².

Why it matters

Small effects need very large samples.

Halving the effect you want to detect needs four times the users.
An under-powered test usually shows 'no significant difference' even when the model helps, and the team wrongly concludes it does not.
Duration also matters: always run at least a full week, so every weekday is included.

In the real world: Kirana Kart's experiment tool asks for the baseline and the minimum effect, and refuses to start a test that could not reach 80% power within four weeks.

Visual example

Computed

Baseline 10% of visits add a recommended item. 10% of 1,000,000 daily users in the test, split evenly: 50,000 per group per day.

5% significance, 80% power; minimum one full week

effect to detect	relative change	users per group	days needed
+0.2 points	+2%	352,800	8
+0.5 points	+5%	56,448	7
+1 point	+10%	14,112	7
+2 points	+20%	3,528	7

Detecting a 2% relative change needs 25 times the users of a 10% change.

Running until the result looks significant, or stopping after two days because the numbers look good. Both inflate false wins (see peeking, below).

Choosing the randomisation unit

What it is

What you randomise decides what the test can measure:

User: the default for features users experience over time.
Session or request: more units, faster results, but users see both versions.
Store, city or region: when users affect each other, as in marketplaces.
Time slots (switchback tests): alternate the whole market between versions by hour, for systems like pricing or dispatch.

Why it matters

Users are not always independent.

In a delivery marketplace, a dispatch model in the treatment group grabs riders the control group then cannot have; both groups are affected, and the comparison is biased.
Randomising by region or by time slot removes that interference, at the cost of fewer units and noisier results.

In the real world: Ride-hailing and delivery companies such as DoorDash and Lyft have described switchback experiments for dispatch and pricing, precisely because user-level tests were biased by interference.

User-level tests for marketplace systems. A new ETA or dispatch model can look excellent because it takes resources from the control group, then deliver nothing when launched to everyone.

Guardrail metrics

What it is

Guardrails are metrics the change must not harm, checked in every experiment:

user experience: page load time, crashes, errors;
business health: orders, revenue, cancellations, refunds;
trust: complaints, unsubscribes, 'not interested' clicks;
data quality: the experiment's own health checks.

Why it matters

A model can win its primary metric by causing harm elsewhere.

A recommender that raises basket additions by showing more snacks may raise cancellations.
A slower model can lose more engagement through latency than it gains through relevance.
Guardrails catch these before launch.

In the real world: Kirana Kart's standard guardrails are p99 page latency, order cancellations, refund requests and app crashes; a breach on any stops the test automatically.

Checking guardrails only for statistical significance. With small groups, a real harm may not be 'significant'; set limits in advance ('cancellations may not rise by more than 1%') and look at the interval.

Peeking and sequential testing

What it is

Peeking is checking the result repeatedly and stopping as soon as it looks significant. It greatly inflates false wins:

random noise crosses the significance line at some point in most tests if you look often enough;
a 5% false-win rate can become 20% or more.

If you need to monitor and stop early, use sequential testing methods designed for it (such as always-valid p-values or group sequential designs), which adjust the threshold for repeated looks.

Why it matters

Teams naturally want to stop early when a result looks good or bad.

Sequential methods allow that safely.
Stopping early for harm (a guardrail breach) is always fine; stopping early for a win needs the adjustment.

In the real world: Experimentation platforms at Optimizely, Netflix and others offer sequential testing specifically because users kept peeking at fixed-horizon tests.

Watching a live dashboard and calling the test on day three because 'it's significant'. Many such wins disappear when rerun.

Practise it: Design Continuous ML Deployment

Novelty, primacy and long-term effects

What it is

User behaviour changes over the course of a test:

novelty effect: people engage with something because it is new, then stop;
primacy effect: people resist a change at first, then adapt;
long-term effects: a change helps this month and hurts in six (showing pushier promotions trains users to ignore them).

Defences: look at the effect over time within the test, run long enough for it to settle, and keep a long-term holdout: a small group kept on the old system for months.

Why it matters

A test's first week may not represent what launch will look like.

Plotting the daily effect shows whether it is stable, fading or growing.
Holdouts measure the combined long-term effect of many launches, which individual tests cannot.

In the real world: Many large companies keep a small percentage of users on a 'holdback' without a quarter's launches, to measure their combined effect at the end of the quarter.

Launching on a week-one lift that is visibly shrinking day by day. The long-term effect may be zero.

Checking the experiment itself

What it is

Before trusting any result, check the experiment is healthy:

sample ratio mismatch (SRM): groups meant to be 50/50 should be very close to 50/50. A significant imbalance means assignment or logging is broken;
A/A tests: run two identical groups; they should show no difference;
exposure: users counted only once they actually saw the model's output.

Why it matters

A broken experiment produces confident nonsense.

An SRM usually means one group loses some users (a crash, a redirect, a logging bug), and those lost users are not random.
A chi-square test on the group sizes catches it in seconds.

In the real world: Microsoft's experimentation team has reported that sample ratio mismatch is one of the most common signs of a broken experiment, and checks it automatically on every test.

Visual example

Computed

A 50/50 test produced 50,600 and 49,400 users. Chi-square = 14.4, far above 10.8 (the 0.1% level): the split is broken.

Chi-square = Σ (observed − expected)² ÷ expected

group	observed	expected	(obs − exp)² ÷ exp
Control	50,600	50,000	7.2
Treatment	49,400	50,000	7.2

Ignoring a small-looking imbalance. 50.6% against 49.4% on 100,000 users is far beyond chance, and means something is wrong.

Variance reduction (CUPED)

What it is

CUPED makes tests faster by removing noise that existed before the test:

measure each user's metric before the experiment (last month's basket additions);
adjust each user's in-test metric by how much their past behaviour predicts it;
compare the adjusted metrics.

Heavy users stay heavy and light users stay light; removing that predictable part leaves the treatment effect clearer.

Why it matters

Less noise means fewer users or shorter tests for the same power.

Reductions of variance by a third or more are common for engagement metrics.
That is the same as running the test on half again as many users, for free.

In the real world: Microsoft introduced CUPED; Booking.com, Netflix and many others use it or a variant as standard on their experiment platforms.

import numpy as np

def cuped(y, x):
    """y: metric during the test; x: the same metric before the test, per user."""
    theta = np.cov(x, y)[0, 1] / np.var(x)
    return y - theta * (x - x.mean())      # same mean, less variance

# Compare cuped(y_treat, x_treat).mean() with cuped(y_ctrl, x_ctrl).mean()

Using a covariate measured during the test. If the treatment can affect it, the adjustment removes part of the real effect. Use only pre-experiment data.

Practise it: Design Continuous ML Deployment

Interleaving for rankers

What it is

Interleaving compares two rankers on the same user:

merge the two rankers' lists into one, alternating picks (team-draft interleaving);
remember which ranker contributed each item;
whichever ranker's items get more clicks or additions wins that impression.

Why it matters

It is far more sensitive than a standard A/B test for ranking.

Each user compares both rankers directly, which removes the large differences between users.
Netflix reported needing about a hundred times fewer users than an A/B test to find the better ranker.
Use it to choose quickly among several rankers, then confirm the winner's business impact with an A/B test.

In the real world: Kirana Kart screens new search rankers with a three-day interleaving test, and A/B tests only the winner.

Using interleaving as the launch decision. It shows which ranker users prefer, not how much revenue or retention changes; that still needs an A/B test.

Bandits versus A/B tests

What it is

A multi-armed bandit shifts traffic towards the better option while the test runs:

start splitting traffic among the options;
as evidence accumulates, give more traffic to those doing better;
keep a little traffic on the others in case they improve.

Why it matters

Bandits trade clear measurement for less wasted traffic.

Good for short-lived choices with fast feedback: which banner, which headline, which promotion this week.
Poor for decisions needing a precise, unbiased effect size, or with slow or long-term outcomes.
Contextual bandits choose per user, using features, and are used for personalisation.

In the real world: News sites use bandits to choose between headline variants within hours; recommendation teams use contextual bandits for exploration, but launch new models through A/B tests.

Visual example

Choice

Use each where it fits.

A/B test or bandit

	A/B test	bandit
Goal	Measure the effect precisely	Earn the most during the test
Traffic split	Fixed	Shifts towards winners
Best for	Model launches; lasting changes	Short-lived choices; fast feedback
Outcome speed needed	Any	Fast (clicks, not 90-day churn)
Result	Clear effect size with an interval	A good choice; effect size unclear

Using a bandit to decide a major model launch. It never cleanly measures the effect, and it can lock in an early leader whose advantage was luck.

Interview questions: online experimentation

What it is

The questions interviewers ask about online experimentation, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Almost every ML design interview ends with 'how would you know it worked?'. A crisp experiment design, with guardrails and a sample-size estimate, is what is expected.

Q: How would you A/B test a new recommendation model?

Randomise users by hash into control and treatment, pick a primary metric such as basket additions from recommendations plus guardrails like latency and cancellations, compute the sample size for the smallest worthwhile effect, run at least a full week without peeking, then analyse the interval and slices.

Q: How do you compute sample size?

From the baseline rate, minimum detectable effect, significance and power: roughly 2(z_α + z_β)² p(1−p)/δ² per group for a rate. Halving the effect needs four times the users.

Q: What is a sample ratio mismatch?

When the groups' sizes differ from the planned split by more than chance allows, detected with a chi-square test. It signals broken assignment or logging, so the results cannot be trusted.

Q: What is interference and how do you handle it?

When treated units affect control units, as in marketplaces sharing riders. Randomise by region or by time slot (switchback tests) instead of by user.

Q: What is CUPED?

Variance reduction using each user's pre-experiment metric as a covariate; it removes predictable noise so tests need fewer users.

Q: When would you use interleaving or a bandit?

Interleaving to pick the best of several rankers quickly with few users; bandits for short-lived choices with fast feedback. Major launches still go through an A/B test.

Saying 'we'd A/B test it' and nothing more.

Exercise: design and analyse an experiment

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: Data scientists at product companies run this analysis weekly; building it once makes experiment interview questions straightforward.

Your task. Plan an A/B test for a model change, then analyse a simulated result including its health checks.

Steps

Write the hypothesis, randomisation unit, primary metric and three guardrails for Kirana Kart's new search ranker.
Compute users per group for baseline 10% and a minimum effect of +0.5 points, and the days needed at your traffic share.
Simulate the test: per-user conversion with a small true lift, plus a pre-period metric correlated with it.
Analyse with a two-proportion test and a confidence interval; then with CUPED; compare interval widths.
Run a sample-ratio check; then deliberately drop 2% of treatment users and run it again.
Simulate peeking: check daily and stop at the first p < 0.05, over 1,000 simulated A/A tests, and count false wins.

You are done when

Your planned sample size matches the formula.
CUPED gives a narrower interval than the plain analysis.
The sample-ratio check fails when you drop users.
Your peeking simulation shows far more than 5% false wins.

Stretch. Simulate an interleaving test for two rankers and compare how many users it needs against the A/B test.

Practise it: Design a URL Shortener, Design a Demand Forecasting Platform, Design a Large-Scale Classification Platform, Design a Predictive Maintenance System, Design a Real-Time Anomaly Detection System, Design a Recommendation System, Design an Image Search System, Design an ML Training Pipeline, Design Feature Monitoring, Design Model Monitoring, Design a Model Serving Platform, Design a Personalized News Feed, Design a Real-Time Fraud Detection System, Design a Search Ranking System, Design an Ad Ranking System, Design an ML Feature Store, Design Continuous ML Deployment