Interview: Beginner Questions

Q: What is the difference between an ML model and an ML system?

The model is one learned function; the system is everything that makes it useful: data pipelines, labels, features computed the same way in training and serving, training and release, serving, monitoring and retraining.

Q: Describe the ML lifecycle.

Scope the problem and metric, collect and label data, build a baseline and then a model, deploy safely, then monitor and retrain, looping back as the world changes.

Q: Why do ML systems fail silently?

A wrong prediction looks like a right one and raises no error, so system health can be perfect while quality falls. You have to monitor inputs, predictions and outcomes.

Q: When should you not use ML?

When simple rules work, there are no labels, mistakes are unacceptable, or the problem is too small to justify ongoing upkeep.

Q: What is a baseline and why build one?

The simplest thing that could work, such as a rule or a popularity list. It sets the bar for any model, often delivers much of the value, and becomes the fallback.

Q: What is the difference between online and batch prediction?

Batch computes predictions in advance on a schedule and stores them; online computes them when requested using the latest context. Batch is cheaper; online is fresher.

Q: Where do labels come from?

Natural outcomes (actual delivery time), implicit feedback (clicks, purchases), explicit feedback (ratings) and human annotation, each trading volume, accuracy, cost and delay.

Q: What is label delay?

The time between a prediction and knowing the right answer, such as weeks for chargebacks. Recent examples have incomplete labels, so training and evaluation must account for it.

Q: What should you log for an ML feature?

Each prediction with a request ID, the context, everything shown with positions, the model version and ideally the served features, plus later user actions joined by request ID.

Q: Why log impressions and not only clicks?

Items shown and not chosen are the negative examples; without them you cannot learn preferences or correct for position.

Q: What is data leakage?

A feature that would not be known at prediction time, or that encodes the label. It inflates offline scores and fails in production.

Q: How do you get good human labels?

Clear written guidelines with hard examples, trained annotators, gold questions, overlap between annotators and measured agreement.

Q: What is a feature?

An input to the model, computed by code that must produce the same value at serving time as in training, from information available at that moment.

Q: Give examples of batch, streaming and on-demand features.

Batch: a customer's orders in 90 days. Streaming: payments by this card in the last 10 minutes. On-demand: distance from store to address, computed from the request.

Q: How do you handle a categorical feature with millions of values?

Learned embeddings or the hashing trick, with a reserved value for unknowns.

Q: What is training-serving skew?

A difference between features in training and in production, from different code, defaults, stale data or time leaks. It makes live performance worse than offline.

Q: What is a feature store?

A shared system with feature definitions, an offline store of history for training, an online store of latest values for serving, and point-in-time correct training sets.

Q: What does point-in-time correct mean?

Each training example uses feature values as they were at its prediction time, never later ones.

Q: What model would you start with for tabular data?

A baseline, then logistic or linear regression, then gradient-boosted trees, which usually give the best accuracy for the cost on tables.

Q: When would you use deep learning?

For images, text, audio, sequences, huge ID spaces needing embeddings, or very large data with complex interactions.

Q: How do you detect overfitting?

Compare training and validation performance; a large gap means memorisation. Fix with more data, regularisation or a simpler model.

Q: Why split data by time?

Production predicts the future from the past; random splits leak future information and hide drift.

Q: What is experiment tracking?

Recording each training run's parameters, metrics, data version, code commit and artefacts, so runs can be compared and reproduced, for example with MLflow.

Q: What is a model registry?

A store of model versions with lineage, scores, stage and owner; serving loads approved versions from it, and rollback is a pointer change.

Q: Why not use accuracy for fraud detection?

Fraud is rare; a model that never flags fraud is over 99% accurate and useless. Use precision, recall and PR-AUC.

Q: What are precision and recall?

Precision: of flagged cases, the share that were right. Recall: of real cases, the share that were caught.

Q: How do you choose a threshold?

From the cost of each kind of mistake, picking the threshold with the lowest total cost, then checking constraints like review capacity.

Q: What metric would you use for a ranking problem?

NDCG at k for the final list, recall at k for retrieval, or MRR when one right answer matters.

Q: What is an A/B test?

Randomly splitting users between the old and new model and comparing a pre-chosen metric, with enough users and time to detect the effect.

Q: Why might a model win offline and lose online?

Proxy metrics, data shaped by the old model, training-serving skew, user reactions and latency.

Q: What happens if the model service fails?

A fallback serves a simpler answer (cached result, rule or popular items) so the product keeps working; track the fallback rate.

Q: What is a canary release?

Sending a small share of traffic to the new model first, watching it, and raising the share in stages with automatic rollback if it misbehaves.

Q: What is shadow deployment?

The new model scores live traffic without its outputs being used, to check behaviour and latency safely.

Q: What would you monitor for a deployed model?

System health, input data quality and drift, prediction distributions and accuracy as labels arrive, and the business outcome.

Q: What is data drift?

A change in the distribution of model inputs over time, measured for example with PSI.

Q: What is concept drift?

A change in the relationship between inputs and outcome, so the same inputs now mean something different; it needs retraining.