Pandas · Module 16 of 22 · 5 concepts · about 5 min

Datetime and Time-Series Data

Dates as a type rather than as text, and the operations that only work once they are.

At work: Anything measured over time: daily sales, hourly traffic, delivery times, subscription start and end dates.

to_datetime

What it is

Parses strings to datetime64, with format= for an explicit pattern, errors="coerce" to turn failures into NaT, and utc=True to normalise timezones.

Why it matters

Nothing else in this module works until the column is a real datetime; comparison and sorting on date strings are lexicographic and wrong.

In the real world: Converting order timestamps from two systems (one in IST, one in UTC) into proper dates so daily totals aren't split across the wrong days.

df["ordered_at"] = pd.to_datetime(
    df["ordered_at"], format="%Y-%m-%d %H:%M:%S", errors="coerce", utc=True
)
df["ordered_at"].isna().sum()    # how many failed to parse

Relying on inferred parsing, which is slow on large files and ambiguous between day-first and month-first. 01/02/2024 parses two different ways and neither raises.

Practise it: Keep Only Each User's Latest Row, Monthly Active Users, Pull the Month Out of a Timestamp

The .dt accessor

What it is

Exposes components of a datetime column: .dt.year, .dt.month, .dt.dayofweek, .dt.hour, .dt.date, .dt.to_period("M").

Why it matters

Calendar features — day of week, hour of day, month — are among the most predictive and cheapest features available.

In the real world: Pulling out the day of the week from each order to see that weekend sales are 40% higher, and staffing warehouses to match.

df["dow"] = df["ordered_at"].dt.dayofweek       # 0 = Monday
df["hour"] = df["ordered_at"].dt.hour
df["month"] = df["ordered_at"].dt.to_period("M")
df["is_weekend"] = df["dow"] >= 5

.dt.date returns Python date objects in an object column, losing the dtype. Use .dt.normalize() or to_period if you need to keep working with it.

Practise it: Pull the Month Out of a Timestamp, Tidy Up Column Names

Filtering and slicing by date

What it is

With a sorted DatetimeIndex, df.loc["2026-01"] selects a month and df.loc["2026-01":"2026-03"] a range. Without the index, filter with ordinary masks.

Why it matters

Partial-string indexing is concise and readable for exploratory work.

In the real world: Selecting just last quarter's transactions for a quarterly board report.

ts = df.set_index("ordered_at").sort_index()
ts.loc["2026-01"]                 # a whole month
ts.loc["2026-01":"2026-03"]       # inclusive of the end month

df[df["ordered_at"].between("2026-01-01", "2026-03-31")]   # without an index

Slicing an unsorted DatetimeIndex, which either raises or returns something surprising. Sort the index first.

Practise it: Revenue Per Customer From a Messy CSV

resample

What it is

Group-by for time: resample("D").sum() buckets a time-indexed frame to a frequency, with asfreq and agg variants.

Why it matters

It fills the calendar, so a day with no events becomes a zero rather than an absent row — the thing a plain group-by on a date will not do.

In the real world: Turning minute-by-minute website visits into hourly totals for a traffic dashboard.

daily = ts["total"].resample("D").sum()        # gaps become 0-length days
weekly = ts["total"].resample("W-MON").agg(["sum", "count"])
monthly = ts.resample("MS").agg(revenue=("total", "sum"))

Time differences

What it is

Subtracting datetimes gives a timedelta64; .dt.total_seconds(), .dt.days convert it to numbers. diff() gives the gap between consecutive rows.

Why it matters

Time-since-last-event is a standard feature, and session boundaries are defined by a gap exceeding a threshold.

In the real world: Measuring how long each delivery took from order to doorstep, to find which city misses its 30-minute promise.

df = df.sort_values(["user_id", "ts"])
df["gap"] = df.groupby("user_id")["ts"].diff()
df["gap_mins"] = df["gap"].dt.total_seconds() / 60
df["new_session"] = df["gap"] > pd.Timedelta("30min")
df["session_id"] = df.groupby("user_id")["new_session"].cumsum()

Comparing datetimes with different timezone awareness. A tz-aware and a tz-naive value cannot be subtracted and the error is easy to misread.