Offline Evaluation

Offline evaluation scores a model on held-out historical data. It can tell you:

• whether the model predicts past outcomes better than the baseline and the champion;
• where it is weak (which slices, which kinds of case);
• whether it is fast and small enough.

It cannot tell you how users will react to its predictions; only an online experiment can (module 17).

Offline evaluation is fast and cheap, so it is where most decisions are made.

• Minutes per run, no risk to users.
• It filters out bad candidates before they cost anything.
• But history only records reactions to the old model's choices, which limits what it can say about a new one.

For a yes/no decision at a threshold, every case lands in one of four cells:

• true positive: flagged and really fraud;
• false positive: flagged but honest;
• false negative: not flagged but really fraud;
• true negative: not flagged and honest.

Precision = true positives ÷ all flagged: how many flags were right. Recall = true positives ÷ all real frauds: how much fraud was caught.

The two answer the two questions the business asks.

• Precision: how much reviewer time and customer annoyance are wasted?
• Recall: how much fraud slips through?
• Raising the threshold usually raises precision and lowers recall; the right balance depends on what each mistake costs.
• F1 (their harmonic mean) is a single number when both matter equally, which is rarely exactly true in business.

Two summaries of a model across all thresholds:

• ROC-AUC: the chance that a random positive is scored above a random negative. 0.5 is guessing; 1 is perfect.
• PR-AUC (average precision): the area under the precision-recall curve. Its baseline is the positive rate (0.005 for 0.5% fraud), not 0.5.

For rare positives, PR-AUC is the more honest measure.

• ROC-AUC counts true negatives, which are abundant, so a model can score 0.98 while most of its flags are wrong.
• PR-AUC focuses on the positives and the flags, which is what matters for fraud, failures and ads.
• Both measure ranking quality; neither says which threshold to use, and neither checks calibration.

The best threshold minimises the total cost of mistakes:

• put a price on each false positive (review time, an annoyed honest customer, a lost sale);
• put a price on each false negative (the money lost to an uncaught fraud);
• compute total cost = false positives × their cost + false negatives × their cost, at each threshold;
• choose the lowest, then check it against limits such as how many cases reviewers can handle.

It turns an argument into arithmetic.

• Product wants fewer annoyed customers; risk wants less fraud; both agree on prices, and the threshold follows.
• When prices change (a costlier fraud wave), the threshold can be recomputed without retraining.

For predicted numbers (minutes, rupees):

• MAE (mean absolute error): the average size of a miss, in the same units. Easy to explain.
• RMSE (root mean squared error): punishes large misses much more. Use when big misses are much worse.
• Percentiles of error: the 90th or 99th percentile miss shows the tail that customers remember.
• Asymmetric measures: count late and early misses differently, or report the share of orders more than 5 minutes late.

The average hides what customers feel.

• One delivery 12 minutes late generates a complaint; nine within a couple of minutes generate nothing.
• RMSE and tail percentiles expose those large misses.
• Pick the measure that matches the business harm, and report the tail beside the average.

For an ordered list, measure whether the good items are near the top:

• Precision at k: share of the top k that are relevant.
• Recall at k: share of all relevant items that are in the top k.
• MRR (mean reciprocal rank): 1 ÷ the position of the first relevant item, averaged. Good when one right answer is enough.
• NDCG at k: rewards relevant items more the higher they are, supports graded relevance (bought > added > clicked), and is scaled so 1 is the best possible order.

Position matters: an item at position 1 is seen by nearly everyone, at position 10 by few.

• NDCG captures both position and degrees of relevance, which is why it is the standard ranking measure.
• Recall at k suits retrieval; NDCG and MRR suit the final list.

Forecasts are judged over many items and days:

• WAPE (weighted absolute percentage error): total absolute error ÷ total actual. Robust and easy to explain.
• MASE: error relative to a naive forecast (such as 'same as last week'). Below 1 beats naive.
• Pinball (quantile) loss: for forecasts of a percentile, such as the 90th, used to set stock levels.
• Bias: whether forecasts are too high or too low on average.

Percentage error per item (MAPE) breaks on small or zero values, which are common for slow-selling items.

• WAPE weights by volume, so a big miss on a best-seller counts more than on a niche item.
• Bias matters on its own: steady over-forecasting wastes stock even if the error is small.

Backtesting (rolling-origin evaluation) repeats a time split several times:

• train on data up to cut-off 1, test on the period after;
• move the cut-off forward, retrain, test again;
• repeat over many cut-offs and average, or look at the spread.

One time split can be lucky or unlucky.

• A model tested only on a quiet week looks better than it is.
• Backtesting over months, including holidays and promotions, shows how the model really behaves and how variable it is.
• It also mimics retraining exactly as production will do it.

A model that wins offline can lose online. Common reasons:

• the data records the old model's choices: you only see reactions to items the old model showed, so a new model's different picks cannot be judged fairly;
• the measure is a proxy: NDCG on clicks is not revenue;
• skew: live features differ from training features (module 11);
• users react: novelty, learning, changed behaviour;
• latency: a slower model costs engagement that offline scores ignore.

Knowing the reasons tells you which checks to add.

• Counterfactual methods (re-weighting logged data by how likely each item was to be shown) reduce the first problem.
• Replay tests catch skew before launch.
• Only an online experiment settles the rest.

A few rules that prevent most evaluation mistakes:

• compute every measure on a later period than training;
• keep a test set untouched until the final decision;
• evaluate per slice as well as overall;
• report uncertainty: a confidence interval from bootstrap resampling;
• compare with the baseline and the champion on the same rows.

Small evaluation mistakes cause large wrong decisions.

• A difference of 0.3% may be noise; the interval says whether it is.
• Comparing on different rows makes any model look better or worse at random.

The questions interviewers ask about offline evaluation, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.