Pandas · Module 19 of 22 · 5 concepts · about 5 min

Exploratory Data Analysis with Pandas

Interrogating a dataset before modelling it, with the frame itself rather than a plotting library.

At work: The first look at a business question: what is typical, what is unusual, and what moves together.

Distributions

What it is

describe() for the summary, quantile([.01, .5, .99]) for the tails, value_counts(bins=10) to bucket a numeric column.

Why it matters

The mean and standard deviation describe a normal distribution. Most real columns — spend, session length, request latency — are skewed, and the percentiles are what actually characterise them.

In the real world: Looking at how order values are spread shows most are under ₹500 but a few are over ₹50,000, which changes how free shipping should be priced.

df["total"].quantile([.01, .25, .5, .75, .95, .99])
df["total"].value_counts(bins=10).sort_index()
df["total"].skew()      # far from 0 means the mean is misleading

Reporting a mean for a heavily skewed column. The median and the 99th percentile together say far more, and the mean can sit somewhere almost no observation is.

Counts and cross-tabs

What it is

value_counts(normalize=True) gives proportions; pd.crosstab(a, b) counts combinations of two columns, with normalize for row or column shares.

Why it matters

A cross-tab is the fastest way to see whether two categorical variables are related, and to spot a category that only ever co-occurs with one value of another.

In the real world: A table of plan by cancellation reason shows that premium users mostly leave over price and basic users over missing features.

df["status"].value_counts(normalize=True)
pd.crosstab(df["country"], df["status"])
pd.crosstab(df["country"], df["status"], normalize="index")   # row shares

Correlation

What it is

df.corr() gives pairwise correlations, with method="spearman" for rank correlation that does not assume linearity.

Why it matters

Two features correlated at 0.98 are one feature, and knowing that before training explains a lot of later instability.

In the real world: Checking whether customers who use the app more also spend more before investing in app engagement features.

corr = df.select_dtypes("number").corr(method="spearman")
pairs = corr.unstack().sort_values(ascending=False)
pairs[(pairs < 1) & (pairs > 0.95)]     # near-duplicate features

Reading Pearson correlation as 'related'. It measures linear association only: a clean parabola can score near zero, and one extreme outlier can manufacture a strong correlation from nothing.

Outliers

What it is

Find them with quantile bounds or an IQR rule (q1 - 1.5 * iqr, q3 + 1.5 * iqr), then decide — keep, cap, or investigate.

Why it matters

Outliers are frequently data-entry errors or unit mix-ups rather than rare events, and the difference matters for what you should do.

In the real world: A single ₹10,00,000 order from a test account skews the average order value; spotting it keeps the weekly report honest.

q1, q3 = df["total"].quantile([.25, .75])
iqr = q3 - q1
out = df[(df["total"] < q1 - 1.5 * iqr) | (df["total"] > q3 + 1.5 * iqr)]
out[["order_id", "total"]].head(20)     # look before deleting

Deleting outliers as a reflex. In fraud, failure and anomaly work the outliers are the entire signal.

Grouped summaries

What it is

groupby(key).agg(...) comparing a metric across segments, often with a count alongside so small groups are visible.

Why it matters

An average that moves sharply between segments is the start of most useful findings — and a segment of four rows explains most spurious ones.

In the real world: Comparing average delivery time and customer rating per city, with the number of orders shown, so tiny cities aren't over-read.

summary = df.groupby("country").agg(
    n=("order_id", "count"),
    median_total=("total", "median"),
)
summary[summary["n"] >= 30].sort_values("median_total", ascending=False)

Ranking groups by a mean without showing the count. The top of that list is almost always the smallest groups.