Pandas · Module 4 of 22 · 5 concepts · about 5 min

Understanding Your Dataset

The first few minutes with a frame you did not create. Doing this properly prevents most of the bugs in the modules that follow.

At work: The first ten minutes with any new dataset: what is in it, what is missing, and whether it can be trusted.

head, tail, shape, columns

What it is

head/tail show the ends, shape gives (rows, columns), columns lists the names. The cheapest possible orientation.

Why it matters

Row count alone catches a surprising number of mistakes — a join that fanned out, a filter that removed everything, a file that only partly loaded.

In the real world: The first thing an analyst does with a new marketing export: how many rows, which columns, and does it look like what was promised.

df.shape            # (10432, 7)
df.columns.tolist() # ['order_id', 'customer_id', ...]
df.head(3)
df.tail(3)          # the end often differs: totals rows, truncated writes

Judging data quality from head(). Files are frequently sorted, so the first five rows are the least representative sample available.

Practise it: Top N Products by Revenue, The Top Three Per Group, Deterministically

dtypes and info

What it is

dtypes lists each column's type; info() adds non-null counts and memory usage in one view.

Why it matters

It is the fastest way to spot the two most common data problems at once: a numeric column that came in as object, and a column that is mostly missing.

In the real world: Spotting that “revenue” was loaded as text because one row says “n/a”, before it breaks next month's sales report.

df.info(memory_usage="deep")
#  #   Column    Non-Null Count  Dtype
#  0   order_id  10432 non-null  int64
#  1   total     10201 non-null  object   <- should be float; 231 missing

Skipping this and discovering after a group-by that the numbers were strings all along, sorting lexicographically so that 10 came before 9.

describe

What it is

Summary statistics per numeric column — count, mean, std, min, quartiles, max. include="all" adds counts, uniques and top values for non-numeric columns.

Why it matters

Min and max alone catch impossible values: negative ages, prices of zero, timestamps from 1970 where a null was coerced.

In the real world: A quick check on a delivery dataset shows a minimum delivery time of −3 hours, a sign of a timezone bug before anyone builds on it.

df.describe()                       # numeric columns only
df.describe(include="all").T        # every column, transposed to read down
df["total"].describe(percentiles=[.01, .5, .99])

Reading the mean without the count. A mean over a column that is 90% missing describes the 10% that is not, and says nothing about the rest.

sample, and why not head

What it is

sample(n) draws random rows, with random_state for reproducibility.

Why it matters

It is the honest way to eyeball data, because it does not inherit whatever ordering the file arrived in.

In the real world: A log file is sorted by time, so its first rows are all from one morning. A random sample shows what the whole month really looks like.

df.sample(5, random_state=0)      # reproducible: same 5 rows next run
df.sample(frac=0.01)              # 1% of the frame

Practise it: Top N Products by Revenue, The Top Three Per Group, Deterministically

value_counts and nunique

What it is

value_counts() counts occurrences per distinct value, with dropna=False to include missing. nunique() counts distinct values.

Why it matters

It is how you find out whether a supposed identifier is unique, and whether a categorical column has the five values you expected or eleven with typos.

In the real world: Counting orders per payment method shows 60% card, 30% UPI and 10% cash, and that “Card” and “card” are being counted separately.

df["status"].value_counts(dropna=False)
# completed    8921
# cancelled     982
# Completed      98   <- a typo, and a group-by would treat it as its own status
df["order_id"].nunique() == len(df)   # False means duplicates exist

Leaving dropna at its default while investigating missingness, which hides the exact thing you are looking for.

Practise it: Monthly Active Users