Interview: Advanced Questions

Q: How would you measure the true impact of a recommender, not just clicks it captured?

Clicks on recommended items include purchases that would have happened anyway. Use randomised experiments and long-term holdouts that measure total purchases and retention, not clicks on the row.

Q: How would you train a ranker from logged data produced by an older ranker?

Log propensities, weight examples by inverse propensity (with clipping for variance), add exploration traffic, and evaluate with counterfactual estimators before A/B testing.

Q: Your fraud model's chargeback rate keeps falling. Is it working?

Not necessarily: blocked payments never produce chargebacks. Check false declines, appeals and a random review holdout, which gives unbiased outcomes for cases the model would block.

Q: How do you prevent a recommender from narrowing over time?

Exploration with logged probabilities, diversity in re-ranking, debiasing for position and popularity, and monitoring coverage of the catalogue.

Q: When would you use contextual bandits instead of supervised ranking?

When rewards are fast and you must keep learning about many options, such as new content or promotions; they explore while serving, at the cost of harder offline evaluation.

Q: How do you estimate long-term effects from short experiments?

Long-term holdouts, surrogate metrics validated against long-run outcomes, and watching effect trends within the test for novelty or learning.

Q: Why does calibration matter in ad ranking?

pCTR multiplies the bid in eCPM and sets second prices, so miscalibration mis-ranks ads across segments and over- or under-charges advertisers.

Q: How do you keep a frequently updated click model calibrated?

Correct for sampling, recalibrate on recent unsampled traffic per segment, monitor predicted over actual clicks hourly, and block updates that drift outside limits.

Q: How do you combine scores from several models?

Calibrate each to a common probability scale, then combine as expected value or with learned weights; raw scores from different models are not comparable.

Q: A model has great AUC but poor business results. What could be wrong?

Poor calibration where the number is used, a poor operating threshold, a proxy target misaligned with value, or gains concentrated where they do not matter.

Q: How do you set thresholds when costs change over time?

Keep the decision layer separate, recompute the cost-minimising threshold from current costs and capacity, and monitor precision and recall at the operating point.

Q: How would you handle delayed conversions in conversion prediction?

Delayed-feedback models or importance weighting for unfinished windows, training only on matured examples for final labels, and monitoring by cohort age.

Q: How do you train a model with billions of ID embeddings?

Shard embedding tables across machines, use data parallelism for dense layers, hash rare IDs, and consider frequency-based pruning; checkpoint regularly.

Q: How do you update an ANN index without downtime after retraining the embedding model?

Re-embed the whole catalogue with the new model into a new index, validate recall, then swap atomically; never mix vectors from two models.

Q: How would you serve nearest-neighbour search over a billion items?

Compress vectors with product quantisation, shard the index across machines, query all shards in parallel and merge, with per-shard recall measured against exact search on samples.

Q: Distributed training is slower than expected. What do you check?

Data loading and input pipeline throughput, communication overhead between workers, batch size and learning-rate scaling, stragglers, and whether the job needed distribution at all.

Q: How do you decide how fresh an embedding model must be?

Measure decay of retrieval recall against model age, the rate of new items, and the cost of re-embedding; often a full daily rebuild plus incremental additions.

Q: How would you cut serving cost of a large ranking model by half?

Distil into a smaller student, quantise, add a light pre-ranker to reduce candidates, cache item-side computation, and right-size hardware and autoscaling.

Q: Why can a user-level A/B test mislead in a delivery marketplace?

Treatment and control share riders; a model that grabs resources for treatment harms control, inflating the measured gain. Use switchbacks or geographic clusters.

Q: How would you design features that depend on the last five minutes of activity?

Stream events through a processor with sliding windows on event time, write to an online store with timestamps, check freshness at serving, and backfill by replay with identical logic.

Q: The streaming feature pipeline lags during peak hour. What do you do?

Serve defaults or fallbacks for stale features and track it, scale the processor, check for hot keys or skewed partitions, and decide whether that feature truly needs streaming.

Q: How do you evaluate an ETA model when the estimate changes customer behaviour?

Offline accuracy is necessary but not sufficient; switchback tests measure on-time rate, conversion and system load together, because shorter estimates can increase orders and delays.

Q: How do you design for a ten-times traffic spike during a sale?

Load-test at peak, pre-scale, cache and precompute aggressively, degrade gracefully with tiered fallbacks, and protect the payment-critical path first.

Q: How would you handle exactly-once semantics for financial features?

Idempotency keys on events, deduplication, idempotent state updates, and exactly-once processing where the stack supports it, with reconciliation against the source of truth.

Q: How would you design an ML platform for fifty models across ten teams?

Shared data and feature store, orchestrated training with tracking and registry, gated releases with canaries, monitoring with ownership, and a paved-road template; measure time to production and adoption.

Q: Build or buy the feature store?

Buy or adopt open source unless scale or unusual needs make it inadequate; the team's time is better spent on models and shared definitions than on infrastructure.

Q: How do you control ML infrastructure cost?

Cost attribution per team and model, quotas and priorities, spot capacity for checkpointed training, scale to zero, right-sized models and hardware, and removing low-value features and models.

Q: How do you decide which models deserve investment?

By measured business impact per unit of cost and risk, from experiments and holdouts, compared with simpler alternatives.

Q: How do you avoid every team building its own pipeline?

Make the platform path faster than the alternative: templates, one-command release, defaults that include monitoring; then retire bespoke paths gradually.

Q: What would you put in a model governance process?

Model cards, data lineage, validation before release, approval for high-risk uses, audit trails, monitoring with owners, and periodic review.

Q: How would you tune a detector that blurs faces in public imagery?

For recall at a fixed overlap, with a low threshold, because a missed face is a privacy breach and an extra blur is cheap; tile large images so small faces are detectable, and audit outputs to measure misses.

Q: How do text queries find videos with poor titles?

Index transcripts and on-screen text for keyword search, and embed sampled frames into a shared text-video space with a contrastive model fine-tuned on search logs; fuse both with reciprocal rank fusion.

Q: How would you learn item similarity without item attributes?

From co-occurrence in user sessions, with a skip-gram objective; add strong outcomes such as bookings as extra context, and negatives from the same market so the model learns more than geography.

Q: What is Adamic-Adar and why use it for friend suggestions?

A sum over mutual friends of 1 ÷ log(their degree); it discounts mutual friends who know everyone, making it a better signal than raw mutual counts.

Q: Why are event and job recommendations hard for collaborative filtering?

Every item is new and short-lived, so item ID embeddings have little history; content, organiser or employer features and relationship features such as distance carry the model.

Q: How do you evaluate link prediction honestly?

Snapshot the graph at a cut-off, compute all features from that snapshot, and predict edges formed afterwards, never hiding random edges from today's graph.

Q: Can a model be fair by every definition at once?

Generally not when base rates differ: equal selection, equal error rates and calibration within groups conflict. Choose with business, legal and ethical input and monitor all of them.

Q: How would you audit a credit model for bias without using the protected attribute in the model?

Use the attribute only for evaluation (where lawful), compare error rates and approval rates by group, identify proxy features, and test mitigations.

Q: How do you honour a deletion request in an ML system?

Delete from raw data, features, training sets and caches using lineage; exclude from future training; retrain on a schedule so models stop reflecting the data within a known time.

Q: How would you defend a ranking system against manipulation?

Down-weight untrusted signals (new accounts, bursts), detect coordinated behaviour, use hard-to-fake features, filter fraud from training data, and red-team regularly.

Q: What is model extraction and how do you limit it?

Copying a model through many queries; limit with rate limits, returning decisions rather than raw scores, and anomaly detection on query patterns.

Q: How do you measure how much harmful content gets through?

Prevalence sampling: label a random sample of content users actually saw, per policy and language, rather than counting removals.