ML System Design · Module 9 of 47 · 11 concepts · about 13 min

Feature Engineering for Production

What features look like in a live system: the kinds grouped by how fresh they must be, how numbers, categories and huge ID spaces are turned into model inputs, windowed aggregates, crosses, embeddings, and the real cost of every feature you add.

At work: Usually where most of a model's accuracy comes from. Good features in a simple model beat a sophisticated model on poor features, in almost every production system.

A feature is code that runs at prediction time

What it is

In a notebook a feature is a column. In production it is a function that must produce a value for every live request:

from data that exists at that moment;
within the latency budget;
computed exactly the same way as in the training data.

Why it matters

That changes which features are worth having.

A feature that needs a slow join across three databases may be fine offline and impossible live.
A feature that depends on information arriving later (the final basket value) cannot be used before checkout.
Asking 'how will this be computed at serving time?' for each feature prevents most production surprises.

In the real world: Kirana Kart's ETA team keeps a one-line note beside every feature: where its live value comes from, how old it may be, and who owns it.

Designing features in a notebook from the warehouse, with no thought for serving. The best feature turns out to be a 30-day aggregate nobody can compute in 20 ms.

Kinds of features by freshness

What it is

Features fall into four kinds by when they are computed:

Static: rarely change. A product's category, a store's location.
Batch: recomputed on a schedule, usually daily. A customer's orders in the last 90 days.
Near-real-time (streaming): updated within seconds or minutes from an event stream. Orders at this store in the last 15 minutes.
On-demand (request-time): computed from the request itself. Distance from store to address; items in this basket; hour of day.

Why it matters

Each kind has a different cost and freshness.

Static and batch features are cheap: compute once, look up many times.
Streaming features need a stream processor and an online store (module 10).
On-demand features add latency to every request.
A good design uses the cheapest kind that is fresh enough for each feature.

In the real world: Most of the gain in Kirana Kart's ETA model comes from two near-real-time features: the store's current packing queue and riders available nearby.

Visual example

Freshness

Use the cheapest kind that is fresh enough.

Kirana Kart's delivery-time features, by kind

kind	example	computed	how old it can be	cost
Static	Store's floor area; city	On change	Weeks	Very low
Batch	Store's average packing time over 30 days	Nightly	A day	Low
Streaming	Orders waiting at the store now	Every few seconds	Under a minute	High
On-demand	Distance from store to address	On each request	None	Adds latency

Making everything real-time. A customer's average basket value barely changes in an hour; streaming it costs a great deal for no gain in accuracy.

Practise it: Design a Personalized News Feed

Numeric features

What it is

Numbers often need reshaping before a model can use them well:

Scaling: put features on similar ranges (subtract the mean, divide by the spread). Needed by linear models and neural networks, not by trees.
Log transform: for values spread over many orders of size, such as prices or follower counts.
Clipping: cap extreme values, so one strange order of 10,000 items does not dominate.
Bucketing: turn a number into ranges (0–2 km, 2–5 km, over 5 km), useful when the effect jumps at thresholds.

Why it matters

Raw numbers can mislead a model.

One value a thousand times bigger than the rest can swamp the training of a neural network.
Whatever scaling or clipping is used must be stored and applied identically at serving time, with the training statistics, not recomputed live.

In the real world: Kirana Kart stores the scaling statistics inside the model's saved file, so the serving code cannot accidentally use different ones.

Recomputing the mean and spread on live traffic for scaling. The feature then shifts whenever traffic shifts, and the model sees inputs it was not trained on. Freeze the statistics with the model.

Categorical features

What it is

Categories (city, product category, device) must become numbers. The main ways:

One-hot: one 0/1 column per category. Fine for a few dozen categories.
Target encoding: replace each category by the average label for it. Compact, but leaks easily.
Learned embeddings: a short list of numbers per category, learned during training. Standard for neural networks.
Native handling: some boosted-tree libraries (CatBoost, LightGBM) handle categories directly.

Always reserve a value for unknown categories: new cities and products will appear after training.

Why it matters

The right choice depends on how many categories there are.

One-hot on 50,000 products makes 50,000 columns.
Embeddings scale to millions and let similar categories share what they learn.
Unknown values are certain in production; deciding how to handle them is part of the design.

In the real world: Kirana Kart's fraud model handles 'device model' (thousands of values) with native categorical support in LightGBM, and maps any device not seen in training to 'unknown'.

Target encoding computed on the whole training set, including each row's own label. Every row partly sees its own answer, and the model overrates the feature. Compute it out-of-fold (on other rows only).

Huge ID spaces and the hashing trick

What it is

Some categories have millions of values: user IDs, product IDs, search terms. Two common approaches:

A lookup table from ID to embedding, which must grow and be maintained as IDs appear.
The hashing trick: run each ID through a hash function and keep the result modulo a fixed number of buckets. No table to maintain, and new IDs work immediately.

The cost of hashing is collisions: different IDs that land in the same bucket share an embedding.

Why it matters

Hashing keeps memory fixed and handles new IDs, at the price of some confusion between IDs.

More buckets mean fewer collisions and more memory.
Collisions matter less for rare IDs (which the model knows little about anyway) and more for frequent ones.

In the real world: Large ad and recommendation systems routinely hash billions of IDs and feature combinations into fixed-size tables, sizing the table so that frequent IDs rarely collide.

Visual example

Computed

1,000,000 user IDs hashed into a fixed number of buckets. Share of IDs that share a bucket with at least one other ID.

1 − (1 − 1/buckets)^(IDs − 1)

buckets	buckets per ID	IDs sharing a bucket
1,048,576	1.0	61.5%
4,194,304	4.2	21.2%
16,777,216	16.8	5.8%

Each fourfold increase in the table cuts collisions about three- to fourfold, for four times the memory.

Choosing far fewer buckets than IDs. With one million IDs in a million buckets, most IDs share a bucket with another, and the model cannot tell them apart.

Aggregates over time windows

What it is

Many of the strongest features summarise recent history over a window:

counts: orders in the last 7 days; payments with this card in the last hour;
sums and averages: amount spent in 30 days; average delivery time at this store this week;
rates: share of this customer's orders refunded;
recency: minutes since the last order.

Using several windows (1 hour, 1 day, 30 days) lets the model compare short-term behaviour with the long-term normal.

Why it matters

Behaviour over time is often more predictive than any single fact.

A card used ten times in ten minutes is suspicious; the same card used ten times in a month is not.
Comparing short and long windows ('spent 5× the usual today') captures change.

In the real world: Velocity features ('transactions with this card in the last 5 minutes, hour and day') are among the most important in nearly every card-fraud model.

-- Point-in-time correct: only orders strictly before each prediction.
SELECT
  p.payment_id,
  COUNT(o.order_id)                       AS orders_last_7d,
  COALESCE(SUM(o.amount), 0)              AS spend_last_7d,
  MAX(o.created_at)                       AS last_order_at
FROM payments p
LEFT JOIN orders o
  ON  o.customer_id = p.customer_id
  AND o.created_at <  p.created_at
  AND o.created_at >= p.created_at - INTERVAL '7 days'
GROUP BY p.payment_id;

Computing the window up to the end of the day in training, but up to 'now' in production. Training then sees events that happened after the prediction (a leak), and the live feature never matches.

Practise it: Design a Predictive Maintenance System

Feature crosses

What it is

A feature cross combines two or more features into one, so a model can learn their joint effect:

city × hour of day: rush hour means different things in different cities;
product category × customer segment;
store × day of week.

Why it matters

Simple models cannot discover interactions on their own.

A linear model sees 'city' and 'hour' separately; the cross lets it learn 'Bengaluru at 7 p.m.' specifically.
Trees and neural networks learn many interactions themselves, so hand-made crosses matter most for linear models and for very sparse interactions.
Crosses multiply the number of values, so they are often hashed.

In the real world: Google's 'Wide & Deep' model for app recommendations combined hand-chosen crosses (the wide part, for memorising specific pairs) with a neural network (the deep part, for generalising).

Crossing everything with everything. The number of combinations explodes, most appear once, and the model memorises noise.

Embeddings as features

What it is

An embedding is a list of numbers that places a thing in a space where similar things are close. Embeddings make powerful features:

pretrained text or image embeddings for product titles and photos;
learned embeddings of users and products from past interactions;
similarity between two embeddings (this user and this product) as a single feature.

Why it matters

They bring knowledge the tabular data does not have.

A brand-new product with no sales still has a title and a photo, and so an embedding close to similar products.
Embeddings computed in batch and looked up at serving are cheap to use.

In the real world: Kirana Kart computes an image-and-title embedding for every product nightly; the ranking model uses the similarity between a product's embedding and the average of the customer's recent purchases.

Updating the embedding model without recomputing every stored embedding. Old and new embeddings live in different spaces, and similarities between them are meaningless.

Practise it: Design a Recommendation System

Every feature has a cost

What it is

A feature is not free once it helps accuracy. Each one adds:

latency: another lookup or computation per request;
upkeep: a pipeline to run, monitor and fix;
risk: another input that can break, drift or leak;
dependencies: another team's data the model now relies on.

Remove features that add little. Regularly check which ones the model actually uses.

Why it matters

Unused features are a liability with no benefit.

A feature whose source silently breaks can damage predictions even if it was barely helping.
Fewer features make a model faster, easier to explain and easier to debug.
Feature importance and ablation (retrain without it, compare) show which earn their keep.

In the real world: A review of Kirana Kart's fraud model finds 60 of its 300 features contribute almost nothing; removing them cuts feature-fetch time by a third with no measurable loss.

Adding every feature that improves the offline score by 0.1%. A year later the model depends on 400 inputs from 30 pipelines, and every incident takes days to trace.

Interview questions: feature engineering for production

What it is

The questions interviewers ask about feature engineering for production, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Interviewers often say 'list the features you'd use' and then pick one: 'how do you compute that at serving time?'. That follow-up is where this module is tested.

Q: What kinds of features would you use, by freshness?

Static (rarely change), batch (daily aggregates), streaming (last minutes, from an event stream) and on-demand (from the request itself). Use the cheapest kind that is fresh enough for each.

Q: How do you handle high-cardinality categorical features?

Learned embeddings or the hashing trick for millions of values, native categorical handling in boosted trees, or out-of-fold target encoding. Always reserve a value for unknowns.

Q: What is the hashing trick's trade-off?

Fixed memory and immediate support for new IDs, at the cost of collisions between IDs in the same bucket. More buckets mean fewer collisions and more memory.

Q: Why are time-window aggregates so useful?

They capture recent behaviour and change, such as five payments in ten minutes against a monthly normal. They must be computed only from events before the prediction time.

Q: How do you decide whether to keep a feature?

By its contribution (importance and ablation) against its cost in latency, upkeep and risk. Remove features that add little.

Q: How do embeddings help with new items?

A new product has no history but has a title and image, whose embeddings place it near similar products, so the model can score it from day one.

Listing fifty features with no word on freshness or how they are served.

Exercise: engineer and cost a feature set

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: Feature reviews like this are a routine part of model reviews at mature ML teams; a feature without a named live source does not ship.

Your task. Design features for Kirana Kart's delivery-time model and decide, for each, how it is computed live and whether it is worth its cost.

Steps

List at least twelve candidate features across the four freshness kinds.
For each, write its live source, how old it may be, and its rough cost in milliseconds per request.
Write the point-in-time correct SQL for two windowed aggregates.
Using any public delivery or taxi dataset, train a model with all features you can compute, then rank them by importance.
Remove the bottom third and retrain; compare accuracy.

You are done when

Every feature has a stated live source and freshness.
Your window SQL uses only events strictly before the prediction time.
You have accuracy with and without the weakest features, and a decision on which to keep.

Stretch. Hash a high-cardinality column (such as pickup location ID) into three bucket sizes and measure how accuracy changes.