Pandas · Module 22 of 22 · 4 concepts · about 4 min

Exporting and Building Pipelines

Turning a working notebook into something that runs the same way twice.

At work: Handing results on to a dashboard, a finance team or a model in production, the same way every day.

to_csv, to_parquet, to_excel

What it is

to_csv(index=False) for interchange, to_parquet for anything that will be read back by code, to_excel when a human insists.

Why it matters

Parquet preserves dtypes; CSV does not, so every CSV round-trip re-guesses types and is where 'the dates became strings' comes from.

In the real world: Sending finance an Excel file, sharing a CSV with a partner, and saving Parquet for the next automated job: each format for its reader.

df.to_parquet("out/orders.parquet", index=False)
df.to_csv("out/orders.csv", index=False)

pd.read_parquet("out/orders.parquet").dtypes   # unchanged
pd.read_csv("out/orders.csv").dtypes           # re-guessed

Writing CSV with the default index=True and reading it back later with an unexpected Unnamed: 0 column.

Method chaining

What it is

Composing a transformation as one expression with .assign, .query, .pipe, rather than reassigning a variable at each step.

Why it matters

A chain reads top to bottom as a pipeline, never mutates its input, and makes each step's contribution visible.

In the real world: A cleaning pipeline written as one readable chain of steps is easy to review, and easy to rerun when last month's numbers are questioned.

clean = (
    raw
    .rename(columns=str.lower)
    .query("status == 'completed'")
    .assign(margin=lambda d: d["revenue"] - d["cost"])
    .sort_values("margin", ascending=False)
)

pipe, for your own steps

What it is

df.pipe(fn, *args) inserts a custom function into a chain, so your own transformations compose like built-in ones.

Why it matters

It is what keeps a chain readable once the logic outgrows assign, without falling back to intermediate variables.

In the real world: Plugging the company's own “remove test accounts” step into every report the same way, so no report forgets it.

def drop_test_accounts(d, prefix="test_"):
    return d[~d["email"].str.startswith(prefix, na=False)]

clean = raw.pipe(drop_test_accounts).assign(margin=lambda d: d["revenue"] - d["cost"])

Reproducible transformations

What it is

A pipeline should be a function from raw input to output: no hidden state, no in-place mutation of a frame defined elsewhere, a fixed random_state wherever sampling happens.

Why it matters

Reproducibility is what makes a result reviewable. If rerunning the notebook produces different numbers, nobody can check the work — including you.

In the real world: A monthly revenue report that gives exactly the same answer when re-run next year, which auditors and investors expect.

def build_features(raw: pd.DataFrame) -> pd.DataFrame:
    return (
        raw.pipe(clean)
           .pipe(add_calendar_parts)
           .pipe(add_rolling_spend)
    )

features = build_features(pd.read_parquet("raw.parquet"))

Cells that depend on having been run in a particular order, mutating df in place. It works until the kernel restarts.