Interview: Intermediate Questions

Q: How do you choose between click, add-to-basket and purchase as a target?

Trade signal volume against closeness to the goal: clicks are plentiful but reward curiosity; purchases are close to revenue but rarer and noisier. Add-to-basket is a common middle, or predict several with a multi-task model.

Q: What is a guardrail metric? Give examples.

A metric that must not get worse while optimising the main one: latency, cancellations, complaints, organic engagement. It catches wins that cause harm elsewhere.

Q: How would you connect a model metric to a business metric?

Show offline that the model metric improves, then run an A/B test on a product metric that moves quickly, with the business metric and guardrails measured, and track offline-online agreement over many launches.

Q: When would you predict a percentile instead of a mean?

When errors are asymmetric, such as delivery promises where lateness costs more; quantile regression gives a promise that is exceeded a chosen share of the time.

Q: Why separate the prediction from the decision?

So thresholds and business rules can change without retraining, policy stays readable and auditable, and capacity constraints are applied in one place.

Q: How do you decide batch versus online for a new feature?

Ask whether the prediction depends on request-time context and whether freshness measurably improves it; otherwise batch is cheaper and safer. Often a hybrid: precompute heavy parts, compute light parts live.

Q: You downsampled negatives to 10%. What changes?

Ranking is unaffected but probabilities are inflated; correct with p = qw / (qw + 1 − q) before using them as probabilities, then check calibration.

Q: How would you correct for position bias?

Estimate each position's examination probability with randomised swaps or interventions, then weight training examples by the inverse, or include position as a training-only feature set to a constant at serving.

Q: What is selection bias in a lending model?

Outcomes are only observed for approved applicants, so the model learns a distorted population; mitigate with a random holdout, reject inference or importance weighting.

Q: Why group splits by user?

So the model cannot recognise users it saw in training; otherwise the test score measures memory of known users and overstates performance on new ones.

Q: How do you evaluate a model fairly across segments?

Stratify the test set, ensure enough examples per slice, report per-slice metrics with intervals, and treat a regression in an important slice as a blocker.

Q: How does a feedback loop form in a recommender?

The model shows some items more, they get more clicks, and the next model learns to show them even more. Exploration, propensity logging and diversity monitoring counter it.

Q: Your model was great offline and worse from day one in production. Why?

Most likely training-serving skew or leakage: run a replay test comparing logged served features with recomputed ones, and check that every feature is available at prediction time.

Q: How do you build training data without leaking the future?

Point-in-time as-of joins: for each example, the latest feature value at or before its timestamp, with label windows closed before the example is used.

Q: How do you make pipeline steps safe to re-run?

Partition by date, overwrite the partition, read only that partition's inputs and never use the current time inside a step.

Q: How do you backfill a new streaming feature?

Replay historical events through the same logic by event time, or recompute in batch with identical logic, then verify against logged live values.

Q: What are event time and processing time, and why does it matter?

When it happened versus when it arrived; windows must use event time with watermarks, or delays create false spikes and misplaced events.

Q: When is a feature store worth it?

When several models share features, need low-latency online features, or teams keep hand-writing point-in-time joins; not for a single batch model.

Q: Why use a two-stage recommender?

The accurate ranker is too slow for millions of items, so fast retrieval picks hundreds of candidates and the ranker orders them; retrieval is judged on recall, ranking on order.

Q: How is a two-tower model trained and served?

User and item towers map features to vectors trained so interacting pairs score high, usually with in-batch negatives; item vectors are precomputed and indexed, and only the user tower runs per request.

Q: What are the trade-offs of HNSW versus IVF-PQ?

HNSW is fast with very high recall but memory-hungry; IVF with product quantisation uses far less memory for billion-scale catalogues at some cost in recall.

Q: Pointwise, pairwise or listwise ranking?

Pointwise is simple and calibrated; pairwise and listwise focus on order at the top and can optimise NDCG-like goals but do not give probabilities.

Q: How do you balance relevance and revenue in ranking?

Predict each goal, combine after normalising or as expected value with business weights, cap commercial items in re-ranking, and tune weights by experiment with guardrails.

Q: How do you handle cold-start items?

Content features in the item tower, guaranteed exploration impressions and popularity-based candidate sources.

Q: How many users do you need for an A/B test?

From baseline rate, minimum detectable effect, significance and power; about 2(z_α + z_β)² p(1−p)/δ² per group for a rate, so halving the effect quadruples the users.

Q: What is peeking, and why is it a problem?

Checking results repeatedly and stopping when significant; it inflates false wins well above 5%. Use fixed horizons or sequential tests.

Q: What is a sample ratio mismatch?

Group sizes that differ from the planned split by more than chance; it signals broken assignment or logging and invalidates the result.

Q: When would you randomise by region or time instead of user?

When users interact through shared resources, as in marketplaces; switchback or cluster designs avoid interference bias.

Q: What does CUPED do?

Uses each user's pre-experiment metric to remove predictable variance, so tests reach the same power with fewer users.

Q: What is interleaving and when is it useful?

Merging two rankers' results for the same user and seeing whose items are chosen; very sensitive for picking among rankers, but not a measure of business impact.

Q: How would you reduce inference latency by 3×?

Profile first; then batch feature reads and model calls, shrink the model (fewer trees, quantisation, distillation), compile it, and cache or precompute repeated work.

Q: What is dynamic batching and its trade-off?

Grouping requests from different users into one model call for throughput, especially on GPUs, at the cost of up to the maximum wait in latency.

Q: How do you choose between CPU and GPU serving?

Compare cost per thousand predictions at real traffic including minimum replicas; GPUs win for large neural models at high steady load, CPUs often for trees and modest traffic.

Q: What checks must a model pass before release?

Beat the champion on the same recent data, no important slice regressing, calibration, latency and size limits, and serving checks that it loads and matches offline predictions.

Q: How do you release a model that needs a new feature?

Ship and verify the feature first, unused; then the model; retire old features only when no deployable version needs them.

Q: What are the MLOps maturity levels?

Manual (notebooks, hand deployment), automated continuous training with gates, and automated delivery where the pipelines themselves are tested and released through CI/CD.