Pandas · Module 10 of 22 · 5 concepts · about 5 min

Data Types and Type Conversion

The dtype decides speed, memory and semantics. Most silent wrongness starts here.

At work: Making sure prices are numbers, dates are dates and IDs stay IDs, so sums and joins give the right answer.

Why object is a warning sign

What it is

object means the column holds arbitrary Python objects, usually strings — or a mix.

Why it matters

An object column of numbers is many times slower than an int column, will not compare numerically, and sorts lexicographically.

In the real world: An “amount” column showing as object usually means some values are text like “₹1,200”, so totals silently fail or come out wrong.

s = pd.Series(["10", "9", "100"])     # object
s.max()            # '9'  — lexicographic, not numeric
s.astype(int).max()  # 100

A single stray non-numeric value makes the entire column object, and every downstream numeric operation quietly does the wrong thing rather than failing.

astype

What it is

Casts a column to a given dtype. Strict: it raises if a value cannot be converted.

Why it matters

Being strict is the point. It turns bad data into an exception at the moment of conversion rather than a wrong number later.

In the real world: Storing customer IDs as text so a leading zero isn't lost (“00123” stays “00123”), which would otherwise break joins with the CRM.

df["count"] = df["count"].astype("int64")       # raises on bad values
df["sku"] = df["sku"].astype("string")         # not object
df["tier"] = df["tier"].astype("category")

astype(int) on a column containing NaN raises, because NaN is a float with no integer representation. Clean or use a nullable type first.

to_numeric and to_datetime

What it is

Converters with an errors argument: raise (default), or coerce, which turns unconvertible values into NaN/NaT.

Why it matters

errors="coerce" combined with an isna() count is the standard way to find out how bad a column is before deciding what to do.

In the real world: Turning a messy “price” column from a supplier's feed into numbers, with bad entries marked missing instead of crashing the nightly job.

parsed = pd.to_numeric(df["amount"], errors="coerce")
bad = parsed.isna() & df["amount"].notna()
print(bad.sum(), "unparseable values:", df.loc[bad, "amount"].unique()[:5])

Reaching for coerce to make an error go away. It converts a loud failure into silent data loss unless you check what it discarded.

Practise it: Keep Only Each User's Latest Row, Monthly Active Users, Pull the Month Out of a Timestamp

Nullable dtypes

What it is

Int64, boolean and string (capitalised) hold missing values natively via pd.NA, unlike their NumPy-backed lowercase equivalents.

Why it matters

They let an integer id column keep integer semantics while still having gaps, which is exactly the case that forces the float upcast otherwise.

In the real world: Keeping “number of children” as whole numbers while allowing blanks, instead of it turning into decimals like 2.0 in a customer profile.

pd.Series([1, None]).dtype              # float64 — id becomes 1.0
pd.Series([1, None], dtype="Int64")     # 1, <NA> — still an integer

Categorical dtype

What it is

category stores repeated values once and refers to them by code, optionally with an explicit ordered set of allowed values.

Why it matters

On a column of a few distinct values repeated across millions of rows the memory saving is large, and group-bys get faster.

In the real world: Storing a “country” column with 50 values across 10 million orders as categories cuts memory use sharply and speeds up grouping.

tiers = pd.CategoricalDtype(["low", "medium", "high"], ordered=True)
df["tier"] = df["tier"].astype(tiers)
df[df["tier"] > "low"]     # ordered comparison now means something

Setting categories from training data only, then meeting an unseen value at inference and getting NaN rather than an error.