ML System Design · Module 11 of 47 · 11 concepts · about 14 min

Feature Stores and Training-Serving Skew

The system that keeps features consistent between training and serving: offline and online stores, point-in-time correct training data, a registry for reuse, and the many ways training and production quietly disagree, with how to detect and prevent each.

At work: Shared infrastructure on most mature ML platforms. Training-serving skew is among the most common causes of a model that tested well and underperforms live.

What a feature store is

What it is

A feature store is a shared system for defining, computing, storing and serving features. It has five parts:

a registry: each feature's definition, owner, source and version;
an offline store: full feature history, for building training data;
an online store: the latest value per entity, for fast lookup at serving time;
materialisation jobs that compute features and fill both stores;
APIs: one to build point-in-time correct training sets, one to fetch features at serving time.

Why it matters

It solves three problems at once.

Consistency: training and serving read the same definition, so values match.
Correctness: training data uses only values known at each past moment.
Reuse: a feature built for fraud can be found and used by the ETA team, instead of being rebuilt slightly differently.

In the real world: Uber's Michelangelo platform popularised the idea; today open-source Feast and products such as Tecton, Hopsworks, Databricks and the major clouds' ML services offer one.

Visual example

Feature store

Training and serving read values produced by the same code.

One definition, two stores, two uses

Define onceOwner and version in the registry
MaterialiseBatch and streaming jobs
two stores
Offline storeEvery past value
Online storeLatest value per key
two uses
TrainingPoint-in-time join
ServingMillisecond lookup

Thinking a feature store is just a database. A key-value store holds the latest values; the hard parts are the shared definitions and the point-in-time training sets.

Practise it: Design a Recommendation System, Diagnose Training-Serving Skew, Design a Real-Time Fraud Detection System, Design an ML Feature Store

Offline and online stores

What it is

The two stores serve very different needs:

Offline store: huge, cheap, slow. All history, with timestamps. Read in bulk by training jobs. Usually a warehouse or lakehouse table.
Online store: small, fast, more expensive per byte. Only the latest value per entity. Read one key at a time in milliseconds. Usually Redis, DynamoDB, Cassandra or Bigtable.

Why it matters

No single database does both well.

Training needs to scan billions of historical rows.
Serving needs one row in two milliseconds, thousands of times a second.
The feature store keeps them in step, so the latest value online equals the newest row offline.

In the real world: Kirana Kart keeps online features only for customers active in the last 90 days, about a fifth of all accounts; others get defaults, which cuts the online store's size fivefold.

Putting every feature online. Features used only in batch scoring or analysis belong offline only; the online store's memory is the expensive part.

Practise it: Design an ML Feature Store, Design Continuous ML Deployment

Point-in-time correctness

What it is

When building training data, each example must get the feature value as it was at the moment of that prediction, never a later one. This is done with an as-of join:

for each training example (an entity and a timestamp),
find the feature value for that entity with the latest timestamp at or before the example's timestamp.

Why it matters

Any later value is a leak from the future.

Joining on 'today's value' or 'the nearest snapshot' gives training rows information the live model will never have.
Offline scores look better, and production scores are worse.
Getting this right by hand in SQL is fiddly, which is a main reason feature stores exist.

In the real world: A team's ETA model loses a third of its offline advantage once its training join is changed from 'nearest snapshot' to 'as-of', and the new, lower score turns out to match production exactly.

Visual example

Computed

Orders waiting at store 118, snapshotted every 5 minutes: 18:00 → 12, 18:05 → 20, 18:10 → 31, 18:15 → 18. Only the as-of join gives each order a value it could really have known.

The value each join attaches to each order

order placed	as-of (correct)	nearest snapshot	latest value
18:03	12	20	18
18:09	20	31	18
18:14	31	18	18

The nearest snapshot is sometimes in the future; the latest value always is.

# Feast: build training rows with each feature as it was at each order's time.
entity_df = orders[["store_id", "order_ts", "delivery_minutes"]].rename(
    columns={"order_ts": "event_timestamp"}
)
training_df = store.get_historical_features(
    entity_df=entity_df,
    features=["store_stats:orders_waiting", "store_stats:avg_pack_30d"],
).to_df()

Joining features by date only. An order at 9 a.m. gets the store's end-of-day statistics, which include what happened after 9 a.m.

Practise it: Diagnose Training-Serving Skew, Design an ML Feature Store

Materialisation

What it is

Materialisation computes features and writes them to the stores:

batch materialisation runs on a schedule and writes to both stores;
streaming materialisation updates the online store continuously and appends to the offline store;
on-demand features are computed at request time from the request plus stored features, by code the store also runs for training.

Why it matters

Materialisation is where freshness and cost are set.

Each feature declares how often it is refreshed, and the store schedules it.
Writing both stores from one job is what keeps them consistent.

In the real world: Kirana Kart's store-statistics features materialise every five minutes; its customer-history features once a night; distance is on-demand.

Updating the online store by one job and the offline store by another. The two drift, and training sees different values from serving.

Practise it: Design an ML Feature Store

The registry, reuse and versions

What it is

The registry is the catalogue of every feature:

its definition (the code), entity (customer, store), owner and freshness;
which models use it;
its version: a changed definition is a new version, not an edit in place.

Why it matters

Reuse saves effort and prevents subtle duplicates.

Without a registry, three teams build 'customer orders last 30 days' three ways.
Versions protect models: a model trained on version 1 keeps getting version 1 until it is retrained on version 2.

In the real world: After introducing a registry, a company finds 14 slightly different versions of 'customer lifetime value' across its models and consolidates them into two.

Changing a feature's definition in place. Every model trained on the old meaning silently receives the new one at serving time, a skew created by the platform itself.

Practise it: Design an ML Training Pipeline, Design a Model Serving Platform, Design an ML Feature Store, Design Continuous ML Deployment

Training-serving skew

What it is

Training-serving skew is any difference between the data a model was trained on and the data it receives live. Common causes:

different code computing the 'same' feature in training and serving;
different handling of missing values, defaults or unknown categories;
time leaks in training (values from after the prediction);
stale features at serving, when a pipeline lags;
different sources: training from the warehouse, serving from a live service with its own logic.

Why it matters

It is one of the most common reasons a model underperforms after launch, and it raises no error.

The model receives plausible-looking numbers that mean something slightly different from what it learned.
It often affects only some users (new ones, one platform), so averages hide it.

In the real world: Google's ML rules describe a team whose live model underperformed because a feature was computed from a table that was refreshed differently in production; logging the served features revealed it immediately.

Visual example

Skew

All are invisible to ordinary error monitoring.

Causes of training-serving skew, and the fix for each

cause	example	fix
Two codebases	SQL in training, Java in serving	One definition, shared or generated code
Missing values	Median in training, 0 live	One default, stored with the model
Time leak	End-of-day values in training	As-of joins
Stale serving	Stream stalled for an hour	Freshness checks and fallbacks
Different sources	Warehouse vs live service	Log served features; train on them

Blaming drift. A model that is worse from the first day of launch is usually skewed, not drifting; drift is gradual.

Practise it: Design Feature Monitoring, Diagnose Training-Serving Skew, Design an ML Feature Store

Detecting skew

What it is

Three ways to find skew:

Replay test: take live requests whose served features were logged, recompute the features with the training pipeline, and compare value by value.
Distribution comparison: compare each feature's distribution in training data with its live distribution.
Prediction comparison: score the same requests offline and online and compare the predictions.

Why it matters

The replay test is the sharpest tool.

It shows exactly which features differ and for which rows.
Small differences for fast-changing features are expected (the value moved between computations); big differences on stable features are bugs.
Run it before launch and on a schedule after.

In the real world: Kirana Kart runs a replay test every week on 10,000 logged ETA requests; it once caught a time-zone bug that made 'hour of day' five and a half hours off for training rows.

Visual example

Computed

A replay test on 10,000 logged ETA requests. Each feature's share of rows where training would have used a different value.

Logged live value vs value recomputed by the training pipeline

feature	rows compared	rows that differ	share	reading
distance_km	10,000	0	0%	Consistent
store_avg_pack_30d	10,000	12	0.1%	Small; investigate edge cases
orders_waiting_now	10,000	2,150	21.5%	Expected: value changes by the second
customer_orders_90d	10,000	940	9.4%	Bug: a stable feature should match

Comparing only averages. A feature can have the same average in training and serving while being wrong for every new user, with errors that cancel out.

Preventing skew

What it is

The reliable defences, strongest first:

Log and train on served features (module 5): training sees exactly what serving saw.
One feature definition used by both paths, through a feature store or shared library.
Transformations inside the model: put scaling and encoding in the saved model (for example a scikit-learn Pipeline or preprocessing layers), so serving cannot do them differently.
Point-in-time joins for every training set.
Freshness checks with known fallbacks at serving.

Why it matters

Detection finds skew after it has hurt; prevention stops it existing.

Each defence removes one cause from the table above.
Together they make 'the model behaves as it did offline' the default rather than a hope.

In the real world: Teams using TensorFlow often export preprocessing as part of the model graph (TensorFlow Transform), precisely so the serving system receives raw inputs and applies the identical transformation.

Copy-pasting preprocessing from the training notebook into the serving code. It works on day one and drifts the first time anyone edits one copy.

Do you need a feature store?

What it is

A feature store pays off when several of these are true:

several models share features;
models need online features with low latency;
point-in-time training sets are being hand-written, and getting them wrong;
skew incidents have happened.

For one batch model, or a few models with simple features, a well-organised warehouse plus careful SQL is enough.

Why it matters

A feature store is real infrastructure to run.

It adds a system to operate, learn and pay for.
Its value grows with the number of models and teams.
Build or buy: open-source Feast is light and flexible; managed products save operations work.

In the real world: Many companies start with features in warehouse tables and a shared Python library, and adopt a feature store once a third or fourth live model needs online features.

Introducing a feature store for the first model. Get one model working end to end first; the second and third models show which shared parts are worth building.

Practise it: Design a Recommendation System, Diagnose Training-Serving Skew, Design a Real-Time Fraud Detection System, Design an ML Feature Store

Interview questions: feature stores and skew

What it is

The questions interviewers ask about feature stores and skew, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Feature stores and skew come up in nearly every ML platform interview and often in product ML ones as 'your model was great offline and worse online; why?'.

Q: What is a feature store?

A shared system with a registry of feature definitions, an offline store of history for training, an online store of latest values for serving, materialisation jobs that fill both, and APIs for point-in-time training sets and low-latency lookup.

Q: What is point-in-time correctness?

Each training example gets the feature value as it was at the prediction time, found with an as-of join: the latest value at or before that timestamp. Anything later is a leak.

Q: What is training-serving skew and what causes it?

A difference between training data and live inputs: different code paths, different missing-value handling, time leaks in training, stale serving features, or different data sources.

Q: How would you detect skew?

A replay test comparing logged served values with values recomputed by the training pipeline, plus distribution and prediction comparisons. Stable features that differ point to bugs.

Q: How do you prevent skew?

Train on logged served features, use one feature definition for both paths, put transformations inside the saved model, use point-in-time joins, and check freshness at serving.

Q: When would you not use a feature store?

With one model or only batch scoring, where warehouse tables and careful SQL suffice. It earns its cost with many models, shared features and online serving.

Equating a feature store with Redis.

Practise it: Design Feature Monitoring, Diagnose Training-Serving Skew, Design an ML Feature Store

Exercise: point-in-time join and a replay test

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: ML platform teams often ask candidates to explain exactly this join on a whiteboard; having built it once makes the answer easy.

Your task. Build training data with a correct as-of join, show how a naive join leaks, and run a replay test.

Steps

Create a table of feature snapshots (entity, timestamp, value) and a table of labelled events (entity, timestamp, label).
Join them three ways: as-of (pandas merge_asof with direction='backward'), nearest, and latest value.
Train the same model on each and compare validation scores on a later time period.
Simulate a live service that computes one feature slightly differently (for example, fills nulls with 0 instead of the median), log its values, and recompute offline.
Report, per feature, the share of rows that differ.

You are done when

The leaky joins give better offline scores, and you can explain why that is a lie.
The replay report flags the feature with the different null handling.

Stretch. Install Feast locally, define the same features, and build the training set with get_historical_features.

Practise it: Debug a Production ML Model, Design an ML Feature Store