Requirements, Scale and Estimates

Requirements come in two kinds:

• Functional: what the system does. 'Show ten recommended products on the home screen'; 'score every payment for fraud'.
• Non-functional: how well it must do it. Latency, throughput, availability, freshness, cost, privacy.

For ML systems add two more non-functional ones: how fresh the data behind a prediction must be, and how quickly a new model must reach production.

Non-functional requirements decide the architecture.

• 'Recommendations' could be a nightly batch job or a live service; '100 ms, using what the user did in the last minute' decides which.
• Writing both kinds down gives every later decision something to be checked against.

A back-of-the-envelope estimate is quick arithmetic with round numbers to find the size of a problem:

• how many requests a second;
• how much data a day, and in total;
• how many machines, and roughly what it costs.

Being within a factor of two or three is good enough. The goal is to know whether something is a laptop, a server or a data centre.

Scale changes the design.

• 10 requests a second can be served by one small service; 100,000 needs caching, batching and many machines.
• A training set of 1 GB fits in memory; 10 TB needs distributed processing.
• Interviewers expect a few estimates, done aloud, early in the design.

Start from daily volume and convert:

• average requests a second = requests a day ÷ 86,400 (about 100,000);
• peak = average × a peak factor, often 2 to 5 for consumer apps (evening rush, sales);
• for ranking, multiply by the items scored per request: that is the model's real workload.

Design for the peak, not the average.

• A service sized for the average falls over every evening.
• The fan-out multiplier is what surprises people: a few hundred requests a second, each scoring 500 items, is over a hundred thousand model evaluations a second.

Estimate storage from events a day × bytes per event × days kept:

• count each kind of event: items shown, clicks, additions to basket, searches;
• assume a size per event (a few hundred bytes for an ID-heavy log line);
• multiply by how long you keep it.

Logging is where ML data comes from, and it grows fast.

• Impressions (items shown) dominate: every screen shows many items, and each must be logged to know what was not clicked.
• The total decides where data lives: a database, a data warehouse or cheap object storage.

Estimate the training set from the labels you will use:

• positive examples a day (additions to basket);
• negative examples a day (items shown and not added), usually sampled down, because there are far too many;
• × the days of history used;
• × features per row × bytes per feature.

The size decides how training runs.

• Under a few gigabytes: one machine, in memory.
• Hundreds of gigabytes: a big machine reading from disk, or a small cluster.
• Terabytes: distributed training (module 15).
• It also shows why sampling negatives is standard: keeping them all would be ten times bigger and teach little more.

A latency budget splits the time allowed for one request among its steps:

• start from the product's total limit (the whole screen must load in, say, 300 ms);
• give the ML part a share of it;
• divide that share among network, feature fetching, retrieval, the model and post-processing;
• measure each step at the 95th or 99th percentile, not the average: the slow requests are the ones users notice.

The budget decides what is possible.

• If the ranking model has 40 ms for 500 items, a large neural network is out unless it is made faster.
• Feature fetching is often the surprise: ten separate database reads can use the whole budget.
• Every step that is added must take time from another.

To size a prediction service:

• measure how many requests one copy (replica) handles a second at the latency you need;
• divide peak requests a second by that;
• add headroom (often 30% or more) for spikes, slow machines and a copy being replaced;
• round up, and never go below two copies, so one can fail.

Too few replicas and the service slows down or drops requests at peak; too many and you pay for idle machines.

• Measuring one replica under realistic load is the only reliable starting point.
• Autoscaling adds and removes copies as traffic changes (module 19), but it still needs a sensible minimum.

Availability is the share of time a service answers correctly. Each extra 'nine' cuts the allowed downtime by ten:

• 99% allows several days of downtime a year;
• 99.9% allows under nine hours;
• 99.99% allows under an hour.

For ML features the key design question is: what does the product do when the model cannot answer?

An ML feature should almost never take the product down with it.

• If recommendations fail, show popular items.
• If the ETA model fails, show the simple distance-based estimate.
• If fraud scoring fails, decide in advance whether to let payments through (fail open) or hold them (fail closed).
• A good fallback means the model can have lower availability than the product.

Estimate serving cost from the replica count:

• replicas × price per machine-hour × hours in a month;
• divide by requests a month for cost per thousand predictions;
• add training (machine-hours per run × runs a month), storage and data processing.

Compare the result with the value of the feature: extra revenue, fraud prevented, waste avoided.

Cost is a requirement, not an afterthought.

• A model that adds ₹1 lakh a month in sales but costs ₹3 lakh to serve should not ship.
• Cost per thousand predictions makes options comparable: a bigger model, a GPU, or precomputing in batch.

The questions interviewers ask about requirements and estimates, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.