Bezen

Pandas · Module 5 of 22 · 4 concepts · about 5 min

Selecting Data Correctly

Getting exactly the rows and columns you meant. The rules look interchangeable until an index stops matching positions.

At work: Pulling exactly the customers, columns or dates a report or model needs, and nothing else.

Selecting columns

What it is

df["col"] returns a Series; df[["a", "b"]] returns a DataFrame. Attribute access (df.col) works only for names that are valid identifiers and do not collide with methods.

Why it matters

Choosing between Series and one-column DataFrame deliberately avoids a class of shape errors downstream, particularly when passing features to scikit-learn.

In the real world: A dashboard needs only customer ID, plan and monthly spend from a 200-column table. Selecting just those keeps it fast and clear.

Visual example

Selection

The brackets decide the shape: a list of names gives a DataFrame, one name gives a Series.

df[["f1", "f2"]]

list of names → DataFrame (X)

df["target"]

one name → Series (y)

df
idf1f2target
10.49yes
20.73no
30.25yes

The shapes

df[["f1", "f2"]] → (3, 2)

df["target"] → (3,)

Why this matters

scikit-learn expects X as a 2-D table and y as a 1-D Series. Getting the brackets right avoids a whole class of shape errors.

Common misconception

Wrong idea: df["f1"] and df[["f1"]] are the same thing.

Correct idea: one name gives a Series of shape (n,); a list gives a DataFrame of shape (n, k), even with one name in it.

y = df["target"]            # Series, shape (n,)      <- what sklearn wants for y
X = df[["f1", "f2"]]       # DataFrame, shape (n, 2)  <- what it wants for X

Relying on df.count or df.max style attribute access for real columns with those names — you get the method, not the data, and the failure is confusing.

loc versus iloc

What it is

.loc selects by label, .iloc selects by integer position. Both accept row and column selectors: df.loc[rows, cols].

Why it matters

With a default RangeIndex they behave identically, which is exactly why the difference is usually learnt the hard way — after a filter or sort, labels and positions diverge.

In the real world: Looking up a specific customer by their ID (loc), versus taking the first 10 rows of a sorted list to show a top-10 (iloc).

Visual example

Selection

After a sort, labels and positions disagree — .loc follows labels, .iloc follows positions.

Labels

(kept from before the sort)

df, sorted by total
total
790
270
550

df.iloc[0]

position 0

df.loc[7]

label 7

Same row. df.loc[0] raises

Why this matters

With a fresh 0, 1, 2 index the two agree, which is why the difference is usually learnt the hard way — after a filter or a sort.

Common misconception

Wrong idea: .loc[0] means the first row.

Correct idea: .loc[0] means the row labelled 0 — wherever it is, if it still exists. On another frame it quietly returns a different row.

  1. 1Sort or filterRows move; each keeps its original label.
  2. 2`.iloc` countsPosition 0 is whatever row is now first.
  3. 3`.loc` looks upLabel 7 is found wherever it moved to.
recent = df.sort_values("ordered_at").tail(100)
recent.iloc[0]              # first row of this slice
recent.loc[0]               # the row labelled 0 — probably not in here at all
recent.loc[:, ["total"]]    # rows by label, columns by name

Slice semantics differ

What it is

.loc slices include the endpoint; .iloc slices exclude it, following Python.

Why it matters

It is a genuine asymmetry in the API, not a quirk you can reason around, and it is a frequent source of off-by-one errors.

In the real world: Asking for sales “from January to March” by date label includes March; asking for rows 0 to 3 by position does not. Mixing them up drops a month from a quarterly report.

Visual example

Selection

The same slice, two answers: .loc includes the end label, .iloc stops before it.

.loc[2:5] — 4 rows
v
2c
3d
4e
5f
same 2:5
.iloc[2:5] — 3 rows
v
2c
3d
4e

Counting the rows

.loc[2:5] → 2, 3, 4, 5

.iloc[2:5] → 2, 3, 4

Why this matters

Off-by-one errors here do not raise. A window that is one row too long or too short just gives slightly wrong numbers.

Common misconception

Wrong idea: every slice stops before the end, like Python lists.

Correct idea: .iloc does. .loc includes the end, because a label has no 'one after it'.

df.loc[2:5]    # labels 2,3,4,5 — five rows
df.iloc[2:5]   # positions 2,3,4  — three rows

Scalar access with at and iat

What it is

.at[label, col] and .iat[pos, col_pos] read or write a single cell, and are meaningfully faster than .loc/.iloc for that one job.

Why it matters

Worth knowing exists, though needing it in a loop is usually a sign the work should have been vectorised.

In the real world: Correcting one wrong price in a product catalogue, like a laptop listed at ₹1 instead of ₹1,00,000, without touching anything else.

df.at[10, "total"]        # one cell, by label
df.iat[0, 2]              # one cell, by position
df.at[10, "total"] = 0.0  # and it writes