ML System Design · Module 29 of 47 · 11 concepts · about 13 min

Design: Delivery Time (ETA) Prediction

A complete design for the delivery time Kirana Kart shows before an order is placed: what 'accurate' should mean when lateness hurts more than earliness, splitting the journey into stages, live store and rider features, percentile predictions, updating the estimate during delivery, and testing in a marketplace.

At work: Asked as 'design ETA for a ride-hailing or food-delivery app' or 'predict delivery dates for an online store'. It tests regression, real-time features and the link between a prediction and a promise.

Requirements

What it is

Clarify:

goal: estimates customers can trust: fewer late deliveries against the promise, without padding so much that orders are lost;
where shown: the home screen (rough), checkout (the promise), and live tracking after the order;
scale: about 10 million estimates a day, about 350 a second at peak (module 4);
latency: 50 ms at p99, since checkout waits for it;
freshness: must reflect the store's current queue and rider availability within a minute.

Why it matters

The estimate is a promise, so being late is worse than being early.

Customers who receive an order later than promised complain and order less; a few minutes early costs little.
That asymmetry shapes the target, the loss and the metrics.

In the real world: Delivery apps typically measure 'on-time against promise' as a headline metric beside average error.

Optimising average error. A model that is right on average is late on nearly half of all orders.

Framing

What it is

A regression problem with a deliberate bias:

target: minutes from order placed to delivered;
decomposition: packing time + time until a rider picks up + travel time; each stage predicted separately and summed, or one model with stage features;
percentile prediction: predict a high percentile (for example the 80th) of the time, not the average, using quantile regression (pinball loss);
offline metrics: mean absolute error, share of orders late by more than 5 minutes, pinball loss;
online metrics: on-time rate, complaints about lateness, conversion at checkout.

Why it matters

Decomposition helps both accuracy and debugging.

Each stage has different drivers: packing depends on the store queue; pickup on riders nearby; travel on distance, traffic and weather.
When estimates go wrong, stage-level predictions show which part failed.
The percentile chosen is a business decision balancing lateness against padding.

In the real world: Kirana Kart promises the 80th percentile of its predicted delivery time, chosen by experiment as the point where fewer orders were late without losing checkout conversion.

Visual example

Computed

Ten past orders with identical features took 18, 20, 21, 22, 23, 24, 25, 27, 30, 36 minutes. Promising a higher percentile trades lateness for padding.

Pinball loss at the 80th percentile is lowest near that percentile

promised minutes	share delivered late	average minutes early	pinball loss (q = 0.8)
22	60%	0.7	2.78
24	40%	1.6	2.08
27	20%	3.6	1.68
30	10%	6	1.68

27 minutes is late for 2 of 10 orders: the 80th-percentile promise. Pinball loss is equally low anywhere from 27 to 30, between the 8th and 9th fastest orders.

Predicting the median and then adding a fixed buffer. A fixed buffer is too much in quiet hours and too little at the rush; a percentile model adapts the buffer to the situation.

Data and labels

What it is

Every order produces a timeline of events:

order placed, packing started, packed, rider assigned, picked up, delivered;
rider GPS pings along the way;
store events: queue length, staff on shift.

Labels are the actual durations of each stage.

Why it matters

Stage timestamps make decomposition possible and reveal where delays come from.

Label quality needs care: riders may tap 'delivered' early or late; GPS can be noisy; clocks on phones drift.
Censored examples (cancelled orders) have no delivery time; dropping them silently can bias the model towards easy orders.

In the real world: Kirana Kart cross-checks each 'delivered' tap with the rider's GPS position and flags orders where the two disagree by more than a few minutes.

Trusting the 'delivered' tap without checking. If riders tap it on arrival at the building, the label excludes the time to reach the flat, and the model is systematically optimistic for large apartment blocks.

Practise it: Design Model Monitoring, Design a Search Ranking System

Features

What it is

Grouped by what drives each stage:

packing: items in the order, items needing special handling (frozen, fragile), orders waiting at the store now, staff on shift;
pickup: riders free near the store now, riders returning soon, recent pickup delays;
travel: route distance and estimated travel time from a maps service, time of day, day of week, rain now, building type at the address;
history: the store's and the area's recent average stage times (rolling windows).

Why it matters

The live features (queue, riders, rain) matter most at exactly the times the estimate is hardest.

Static features make estimates right on quiet afternoons; live features make them right at 8 p.m. on a rainy Friday.
These are streaming features (module 10) with freshness checks.

In the real world: Kirana Kart's two most important features are 'orders waiting at the store now' and 'riders free within 2 km now', both streamed every few seconds.

Using the rider's eventual route length as a feature. It is only known after delivery, a leak that makes offline error look tiny.

Practise it: Design a Demand Forecasting Platform, Design a Predictive Maintenance System, Design a Real-Time Fraud Detection System, Design an ML Feature Store

Model and architecture

What it is

A practical design:

three gradient-boosted models, one per stage, each trained with a quantile objective, plus a small model that combines them (stage times are not independent: a busy store often has few free riders);
a heuristic fallback: maps travel time + the store's average packing time;
streaming features from the store and rider systems into the online store;
a decision layer: rounding to a range shown to the customer ('25–30 min'), caps, and a manual 'surge' override for operations staff.

Why it matters

Boosted trees fit this tabular, feature-rich problem and serve in about a millisecond.

Stage models make debugging and monitoring per stage possible.
Operations staff need an override for events no model has seen (a road closure).

In the real world: Uber, DoorDash and others have described ETA systems that combine routing-engine estimates with machine-learned corrections, rather than replacing the routing engine.

Visual example

ETA

Stage predictions, combined into a promise.

One checkout estimate

FeaturesOrder, queue, riders, route, weather
three stage models, in parallel
PackingMinutes until packed
PickupMinutes until a rider collects
TravelMinutes from store to door
Combine80th percentile of the total
Promise< 50 msRange shown; logged

import lightgbm as lgb

# One quantile model per stage; the 80th percentile of the total is learned by a combiner.
stage_models = {
    stage: lgb.LGBMRegressor(objective="quantile", alpha=0.8, n_estimators=400)
    for stage in ["packing", "pickup", "travel"]
}
for stage, model in stage_models.items():
    model.fit(X_train[STAGE_FEATURES[stage]], y_train[f"{stage}_minutes"])

combiner = lgb.LGBMRegressor(objective="quantile", alpha=0.8, n_estimators=200)
combiner.fit(stage_predictions(X_train) | context(X_train), y_train["total_minutes"])

Adding the 80th percentiles of the three stages. The sum of percentiles is not the percentile of the sum; it overstates the total. Model the total's percentile directly, or combine properly.

Updating the estimate during delivery

What it is

After the order is placed, the estimate should improve as stages complete:

re-predict at each event (packed, picked up) using the stages already finished;
during travel, use the rider's live position and remaining route;
smooth updates so the estimate does not jump around;
alert support and the customer early if the order is predicted to be badly late.

Why it matters

A live estimate that is honest keeps trust even when things go wrong.

Customers accept a delay they are told about early far more than one they discover at the promised time.
Predicted lateness can trigger action: reassigning a rider, offering a voucher proactively.

In the real world: Kirana Kart sends an apology and a small credit automatically when the live model predicts an order will be more than 15 minutes late, before the customer complains.

Showing a live estimate that never increases, to avoid bad news, then delivering late. Customers learn the number means nothing.

Serving

What it is

Meeting 50 ms at checkout:

one batched read of store, rider and customer features;
a maps call for route time, cached per store and address grid cell for a few minutes;
the stage models and combiner in one process, a few milliseconds in all;
a timeout at 30 ms falling back to the heuristic (module 18);
every estimate logged with its features and model version for training and debugging.

Why it matters

The maps call is usually the slowest part; caching by grid cell removes most of it.

The heuristic fallback means checkout always shows a number.

In the real world: Kirana Kart's home screen shows a store-level estimate refreshed every minute; only checkout computes a per-order estimate.

Calling the maps service live on every home-screen view. It is slow, costly and unnecessary for a rough estimate; cache it.

Practise it: Design Feature Monitoring, Diagnose Training-Serving Skew, Design a Model Serving Platform, Design an ML Feature Store

Evaluation in a marketplace

What it is

Offline and online, with care:

offline: time-split backtests over months; MAE, late-by-5 share and pinball loss; slices by city, store, hour, weather, new stores;
online: the estimate itself changes behaviour (customers may not order if the estimate is long; operations react), and riders are shared across all customers;
so a user-level A/B test can be biased; use switchback tests (alternate whole cities or zones between models by time slot) where interference matters (module 17).

Why it matters

ETA is a marketplace problem: one customer's order affects the riders available to another.

Showing shorter estimates to the treatment group may raise their orders and slow everyone's deliveries, which a user-level test attributes wrongly.

In the real world: DoorDash has described switchback experiments for exactly this reason in delivery-time and dispatch changes.

Visual example

Testing

Choose by whether groups share resources.

User-level A/B test or switchback

	user-level A/B	switchback by zone and hour
Unit	Customer	Zone × time slot
Interference	Treatment and control share riders: biased	Whole zone on one model at a time
Sample size	Large	Smaller; needs more time
Good for	Display changes with no shared resources	ETA, dispatch, pricing

A user-level A/B test showing the new model 'raises conversion' by being optimistic. Launched to everyone, the extra orders overload riders and lateness rises across the city.

Monitoring

What it is

Watch accuracy closely, since labels arrive within the hour:

on-time rate and MAE by city, store and hour, updated hourly;
per-stage error, to see which part is failing;
feature freshness for queue and rider features, and fallback rate;
events: rain, festivals, store openings, which shift times suddenly (concept drift, module 21).

Why it matters

Fast labels make ETA one of the easiest models to monitor and retrain.

Hourly per-store accuracy catches a store problem (a broken scanner, short-staffing) the same evening.
Nightly retraining keeps up with gradual change.

In the real world: Kirana Kart's store operations team receives an hourly list of stores whose packing stage is running far over prediction, which usually signals a staffing problem rather than a model one.

Aggregating accuracy daily across the country. A store that is 20 minutes late every evening barely moves the national number.

Practise it: Design Feature Monitoring, Design Model Monitoring

Interview questions: the ETA design

What it is

The questions interviewers ask about the ETA design, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: ETA interviews often turn to the asymmetry of lateness and to marketplace testing; both separate strong answers from ordinary ones.

Q: Why predict a percentile instead of the average?

Because the estimate is a promise and lateness hurts more than earliness. Quantile regression at, say, the 80th percentile is late about one time in five, with padding that adapts to the situation.

Q: Why split the delivery into stages?

Each stage has different drivers, stage labels exist, and stage-level errors show what is going wrong. Combine stages carefully, since percentiles do not simply add.

Q: Which features matter most?

Live ones: orders waiting at the store, free riders nearby, rain now, plus route time from a maps service and recent stage averages.

Q: How would you A/B test a new ETA model?

Carefully, because customers share riders: use switchback tests by zone and time slot, measuring on-time rate, complaints and conversion.

Q: What is your fallback?

Maps travel time plus the store's average packing and pickup time, served if the model times out or features are stale.

Q: How do you handle cancelled orders in training?

They have no delivery time (censored); keep a record of them, check they do not skew the training set towards easy orders, and consider modelling cancellation separately.

Training on average error and reporting MAE only.

Exercise: build a percentile ETA model

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: This percentile comparison is a favourite discussion in ETA interviews at delivery and mobility companies.

Your task. Train delivery-time models for the median and the 80th percentile on public trip data and compare them as promises.

Steps

Use a public taxi or food-delivery dataset with pickup time, drop-off time and locations.
Build features: distance, hour, day of week, pickup area, and a rolling 'trips started from this area in the last 30 minutes' feature computed only from earlier trips.
Train LightGBM with a squared-error objective, then with quantile objectives at 0.5 and 0.8.
For each model, report MAE, share of trips longer than the prediction, and pinball loss at 0.8, on a later time period.
Compare the 0.8 model with 'median prediction + fixed 5 minutes' by hour of day.

You are done when

The 0.8 model is late on about 20% of trips across most hours.
The fixed-buffer approach is too late at busy hours or too padded at quiet ones, and your table shows which.

Stretch. Add a simulated rain feature that slows trips by 20% on random days, and show how much error falls when the model can see it.