ML System Design · Module 8 of 47 · 12 concepts · about 15 min

Data Pipelines and Data Quality

How raw data becomes clean training tables and features on a schedule, how to make each step safe to re-run, and how to catch bad data before it reaches a model.

At work: The daily plumbing of every ML team. When an ML system misbehaves, the cause is more often a data pipeline than the model.

What a data pipeline is

What it is

A data pipeline is a series of steps that turn raw data into something useful, run automatically on a schedule or when new data lands. Each step:

reads from the outputs of earlier steps;
does one job: clean, join, aggregate, compute features;
writes a result that later steps read.

The steps and their dependencies form a DAG (a directed graph with no loops): each arrow points from a step to the steps that need it.

Why it matters

ML depends on many such steps, and they must run in the right order, every day, without anyone watching.

A training table may depend on twenty upstream tables.
If one is late or wrong, everything after it is too.
Thinking in steps and dependencies makes failures local and fixable.

In the real world: Kirana Kart's nightly training pipeline for delivery times has about a dozen steps, from loading yesterday's orders and GPS pings to writing the finished training table.

Visual example

Pipeline

Each box is a separate, re-runnable step.

Kirana Kart's nightly delivery-time pipeline

LoadOrders, rider GPS, store events
ValidateSchema, counts, ranges
CleanTest orders out; time zones; duplicates
JoinWeather, store load, traffic
FeaturesDistance, hour, queue, rolling averages
LabelActual minutes to doorstep
WriteVersioned training table

One giant script that does everything. When it fails at step 14 of 20, it starts again from step 1, and nobody can test a single part of it.

Orchestration

What it is

An orchestrator runs pipelines for you:

schedules them (every night at 2 a.m., or when a file arrives);
runs steps in dependency order, in parallel where possible;
retries failed steps and alerts someone when retries run out;
keeps a history of every run, its inputs and its logs.

Common choices: Apache Airflow, Dagster, Prefect, and for ML specifically Kubeflow Pipelines or cloud services such as Vertex AI Pipelines and SageMaker Pipelines.

Why it matters

Without one, pipelines become cron jobs that fail silently.

The orchestrator is where you see that last night's training data was never built.
Its run history is how you answer 'which data trained the model now in production?'

In the real world: Airflow, first built at Airbnb, is the most widely used orchestrator for data and ML pipelines; many ML teams now choose Dagster for its focus on the data each step produces.

Putting heavy data processing inside the orchestrator's own workers. The orchestrator should tell a data engine (Spark, the warehouse, a training cluster) what to do, not do the work itself.

Practise it: Design an ML Training Pipeline

Idempotent steps and backfills

What it is

A step is idempotent if running it twice gives the same result as running it once. Build every step that way:

each run processes a fixed partition (one day, one hour), named by its date, not by 'now';
it overwrites that partition's output instead of appending to it;
it reads only inputs for that partition.

Then a backfill (re-running past dates after a fix or a new feature) is safe and simple.

Why it matters

Pipelines are re-run constantly: after failures, bug fixes and new features.

A step that appends will double the data on a re-run.
A step that uses 'today' cannot rebuild last month.
Idempotent, partitioned steps make backfilling a year of a new feature a routine job.

In the real world: When Kirana Kart adds a 'rain in the last hour' feature, it backfills twelve months of daily partitions in an afternoon, because each day's step can be re-run on its own.

-- Safe: rebuild exactly one day's partition, however many times it runs.
INSERT OVERWRITE TABLE features.delivery_daily PARTITION (ds = '{{ ds }}')
SELECT order_id, distance_km, store_queue_len, hour_of_day
FROM clean.orders
WHERE ds = '{{ ds }}';

-- Unsafe: depends on when it runs, and appends duplicates on a re-run.
-- INSERT INTO features.delivery_daily
-- SELECT ... FROM clean.orders WHERE created_at > now() - interval '1 day';

Using the current time inside a pipeline step. A step that computes 'last 7 days' from now() gives a different answer every time it runs, so no past run can ever be reproduced.

Data validation

What it is

Data validation checks data against expectations before it is used:

schema: the expected columns and types are present;
completeness: row counts are in a normal range; required fields are not null;
values: within valid ranges (delivery minutes between 1 and 300), from allowed sets (known city codes);
uniqueness: no duplicate order IDs;
distribution: averages and shares close to recent days.

Tools include Great Expectations, TensorFlow Data Validation, Pandera and dbt tests.

Why it matters

A check that fails loudly is far cheaper than a model that quietly learns from broken data.

Run checks at every step's output, so a failure points to the step that caused it.
Decide per check whether to stop the pipeline (missing column) or warn (a small shift).

In the real world: A row-count check on Kirana Kart's orders table catches a day with 40% fewer rows than usual; an app release had stopped logging orders from one payment method.

import pandera as pa

orders_schema = pa.DataFrameSchema({
    "order_id": pa.Column(str, unique=True),
    "city": pa.Column(str, pa.Check.isin(KNOWN_CITIES)),
    "delivery_minutes": pa.Column(float, pa.Check.in_range(1, 300), nullable=False),
    "basket_value": pa.Column(float, pa.Check.ge(0)),
})

orders_schema.validate(daily_orders)   # raises with a list of every failing row and rule

Validating only the schema. A column can have the right name and type and be entirely zeros because an upstream service started sending defaults.

Practise it: Design an ML Training Pipeline

How data breaks

What it is

The same few failures cause most data incidents:

missing data: a partition never arrives, or arrives half-empty;
late data: events arrive after the pipeline ran;
duplicates: retries or replays write events twice;
schema changes: a renamed, removed or retyped field;
meaning changes: same field, new units or new definition;
default values: a broken service sends 0 or -1 instead of failing.

Why it matters

Knowing the list lets you write a check for each.

Most can be caught with simple rules: row counts, null rates, value ranges, distinct counts.
Meaning changes are the hardest, which is why data contracts (module 5) matter.

In the real world: A retailer's demand forecasts fall apart for a week because a supplier feed changed from cases to individual units; a range check on order quantities would have caught it on day one.

Visual example

Failures

Each check is a few lines of configuration.

Common data failures, and the check that catches each

failure	what it looks like	check
Missing partition	Yesterday has no rows	Row count above a minimum
Late data	Today's count rises after the run	Wait for completeness; re-run late partitions
Duplicates	Same order ID twice	Uniqueness on the key
Schema change	Column missing or retyped	Schema check against a contract
Meaning change	Prices 100× bigger	Range and distribution checks
Silent defaults	Many zeros or -1s	Share of default values below a limit

Treating a model's sudden drop in accuracy as a modelling problem. Check the data first: in practice a broken pipeline is the most common cause.

Data versioning

What it is

Data versioning means you can get back the exact data any model was trained on:

snapshot training tables, or use table formats with time travel (Delta Lake, Iceberg), and record the version used;
or use a tool such as DVC that tracks data files alongside code;
record the data version with every trained model.

Why it matters

Without it, nothing can be reproduced.

'Why is the new model worse?' cannot be answered if the old training data no longer exists.
Audits and regulators may ask what data a decision was based on.
Comparing two models is only fair on the same data.

In the real world: Kirana Kart stores every training table in a lakehouse format and writes the table version into the model registry entry, so any model can be retrained bit-for-bit.

Training straight from tables that are overwritten every night. A month later, the data behind the model in production cannot be recovered.

Practise it: Design an ML Training Pipeline

Lineage

What it is

Lineage records where each piece of data came from and what was built from it:

upstream: which sources and steps produced this table or feature;
downstream: which features, models and dashboards depend on it.

Orchestrators, catalogues (DataHub, Amundsen, Unity Catalog) and standards such as OpenLineage collect it automatically.

Why it matters

Lineage answers two questions you will be asked during every incident.

'This feature looks wrong; where did it come from?' (trace upstream).
'This source was broken for three days; which models are affected?' (trace downstream).
It also makes deletion requests and audits possible.

In the real world: After a bad partition in a payments table, a team uses its lineage graph to find, in minutes, the three models and nine features that read it, and retrains only those.

Keeping lineage in people's heads. When the person who built the pipeline leaves, nobody knows that the fraud model depends on a table someone else plans to delete.

Practise it: Design an ML Training Pipeline, Design Feature Monitoring, Debug a Production ML Model

Freshness and late data

What it is

Freshness is how old data is when it is used. Set a freshness target for each important table or feature, such as 'yesterday's orders complete by 3 a.m.'. For late events:

wait a set time before treating a period as complete;
re-run recent partitions when late data arrives;
record each row's event time (when it happened) separately from its arrival time.

Why it matters

Stale data degrades predictions without any error.

A feature store that stopped updating keeps serving yesterday's values.
Mixing up event time and arrival time places events on the wrong day and breaks time-based features.
Alerts on freshness catch these before users notice.

In the real world: Kirana Kart alerts if the 'store queue length' feature is more than five minutes old, because a stale queue makes every delivery estimate from that store wrong.

Monitoring whether a pipeline ran, not whether its data is fresh. A job can succeed every night while reading a source that stopped updating a week ago.

Practise it: Design an Image Search System, Design Feature Monitoring, Debug a Production ML Model, Design a Personalized News Feed

Missing values, handled the same everywhere

What it is

Missing values must be treated identically in training and in production:

decide what missing means: unknown, not applicable, or zero;
fill it the same way in both places (the same default, or the same 'missing' flag);
keep a separate 'was missing' feature when missingness itself is informative.

Why it matters

This is one of the commonest sources of training-serving skew.

Training data from the warehouse has nulls; the live service fills them with 0, or the other way round.
The model then meets values it never saw in training.
Missingness often carries signal: a new account has no history, which itself predicts fraud.

In the real world: A credit model degrades for new customers after launch; the cause is that training filled missing income with the median while the live service sent zero.

Different teams filling nulls differently. The training pipeline uses the column average; the live service uses -1; the model behaves strangely for every new user and nobody knows why.

Testing data pipelines

What it is

Pipelines are code and need tests like code:

unit tests for each transformation, on small hand-made inputs with known outputs;
integration tests that run the pipeline end to end on a small sample;
data tests (validation checks) on every real run;
review of pipeline changes, like any other code.

Why it matters

A bug in a feature transformation becomes a bug in every model that uses it.

Unit tests catch logic errors before they reach a nightly run.
Data tests catch problems with the inputs, which unit tests cannot.
Both are needed because they catch different failures.

In the real world: A team adds a unit test for its 'orders in the last 7 days' feature using five hand-written orders, and finds the window had been including the current order itself, a small leak worth fixing.

Relying on the model's accuracy as the test. A small error in a feature may lower accuracy by a little, unnoticed, for months.

Practise it: Design an ML Training Pipeline

Interview questions: data pipelines and quality

What it is

The questions interviewers ask about data pipelines and quality, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: MLOps and ML platform interviews spend a whole section here; product ML interviews usually test it with one scenario question about a sudden drop.

Q: What makes a pipeline step safe to re-run?

Idempotence: it processes a fixed partition named by date, overwrites that partition's output, and never depends on the current time. Re-runs and backfills then give the same result.

Q: What would you validate in ML training data?

Schema, row counts, null rates, value ranges, allowed categories, uniqueness of keys, and distribution shifts against recent days, with the serious failures stopping the pipeline.

Q: Why version data?

To reproduce any model, compare models fairly, debug regressions and answer audits. Record the data version with every trained model.

Q: What is lineage used for?

Tracing a bad value upstream to its source, and tracing a broken source downstream to every affected feature and model.

Q: A model's accuracy suddenly drops. Where do you look first?

The data: missing or late partitions, schema or unit changes, a spike in default values, or stale features. Pipeline breaks are more common than real model decay.

Q: What does an orchestrator do?

Schedules pipelines, runs steps in dependency order, retries and alerts on failure, and keeps a history of runs, for example Airflow, Dagster or Kubeflow Pipelines.

Practise it: Diagnose Model Drift, Design an ML Training Pipeline, Design Feature Monitoring, Design Model Monitoring, Design an Ad Ranking System

Exercise: build a validated, re-runnable pipeline

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: Junior ML engineers are often given exactly this as a first task; doing it well earns trust quickly.

Your task. Build a small daily pipeline that turns raw event files into a feature table, safe to re-run and protected by checks.

Steps

Generate or download a week of event files, one file per day (for example public taxi trips or e-commerce events).
Write three steps as separate functions: load and clean one day, compute features for one day, write that day's partition.
Make each step take the date as an argument and overwrite its output for that date.
Add validation after loading: schema, a row-count minimum, value ranges and uniqueness.
Corrupt one day's file on purpose (drop a column, duplicate rows, multiply a value by 100) and run the pipeline.
Run any day twice and confirm the output is identical.

You are done when

Each corruption is caught by a named check with a clear message.
Re-running a day changes nothing in the output.
You can backfill all seven days with one loop.

Stretch. Schedule it in Airflow or Dagster with retries and an alert on failure.