ML System Design · Module 6 of 47 · 12 concepts · about 15 min

Labels

Where the right answers come from, why they arrive late or wrong, how to define them precisely, how to get people to label well, and the cheaper ways to get labels: weak supervision and active learning.

At work: Often the real bottleneck of an ML project. Teams find they can build any model they like, but cannot get enough correct, timely labels to train or judge it.

Where labels come from

What it is

A label is the right answer for one example. It comes from one of four places:

Natural labels: the world reveals the answer. The delivery took 27 minutes; the user bought the item.
Implicit feedback: behaviour that suggests the answer. A click, a skip, time spent reading.
Explicit feedback: the user tells you. A star rating, 'not interested', a reported problem.
Human annotation: people paid to judge examples. Reviewers marking fraud, annotators tagging images.

Why it matters

Each source trades volume, cost, accuracy and delay.

Natural labels are free and accurate but may take weeks.
Implicit feedback is plentiful but ambiguous: a click may be curiosity.
Explicit feedback is rare and comes from unusual users.
Annotation is accurate if done well, but costs money for every label.

In the real world: Kirana Kart's delivery-time model has natural labels within the hour; its fraud model waits up to ninety days for chargebacks; its search ranking uses clicks and additions to the basket.

Visual example

Label sources

No source is best on every column.

Kirana Kart's labels, by source

source	example	volume	accuracy	delay
Natural	Actual delivery minutes	High	High	Under an hour
Implicit	Added to basket after a search	Very high	Medium	Minutes
Explicit	'Not interested' on a product	Low	Medium	Minutes
Annotation	Reviewer marks an account as fraud	Low	High if guided well	Days
Natural, slow	Chargeback from the bank	Low	High	Up to 90 days

Assuming the label is obvious. 'Fraud' might mean a chargeback, a reviewer's decision, or an account later banned; each gives a different model.

Defining the label precisely

What it is

A usable label definition says exactly:

which event counts ('added to basket', not 'liked');
within what window of the prediction ('in the same session', 'within 30 days');
measured from when (the moment of the prediction, not the end of the month);
what happens to edge cases: cancelled orders, returns, users who leave before the window ends.

Why it matters

The definition is part of the model.

'Churned' as 'no order in 30 days' and 'no order in 90 days' give very different models and very different numbers of positives.
A precise window also tells you how long to wait before a training example's label is final.

In the real world: A subscription company finds half its 'churners' come back within 45 days, changes the definition from 30 to 60 days of inactivity, and its retention offers stop going to people who were about to return anyway.

-- One training row per customer per week: did they churn?
-- Label = 1 if the customer places no order in the 30 days after the snapshot date.
SELECT
  c.customer_id,
  s.snapshot_date,
  CASE WHEN COUNT(o.order_id) = 0 THEN 1 ELSE 0 END AS churned
FROM snapshots s
JOIN customers c ON c.customer_id = s.customer_id
LEFT JOIN orders o
  ON o.customer_id = c.customer_id
 AND o.created_at >  s.snapshot_date
 AND o.created_at <= s.snapshot_date + INTERVAL '30 days'
WHERE s.snapshot_date <= CURRENT_DATE - INTERVAL '30 days'   -- only rows whose window has closed
GROUP BY c.customer_id, s.snapshot_date;

Including examples whose window has not closed. A customer snapshotted ten days ago with no order yet is not a churner; they may order tomorrow. Leaving the filter out labels them as churned and teaches the model nonsense.

Label delay

What it is

Label delay is the time between a prediction and its right answer becoming known:

seconds for a click;
an hour for a delivery time;
weeks or months for fraud (chargebacks), loan defaults or churn.

Why it matters

Delay shapes the whole system.

You cannot judge a new model faster than its labels arrive.
The most recent examples have incomplete labels: fraud that has not been reported yet looks like a legitimate order.
Training on them teaches the model that recent fraud patterns are safe, exactly when fraudsters are trying new ones.

In the real world: Payment companies typically train fraud models on transactions at least 60 to 90 days old for final labels, and use faster signals (reviewer decisions, customer complaints) to watch recent behaviour.

Visual example

Computed

True fraud rate 0.5%. Chargebacks arrive slowly, so recent payments look far safer than they are.

Illustrative share of chargebacks arrived by each age of payment

age of payment	chargebacks arrived	fraud rate you would see	fraud labelled 'legitimate'
7 days	30%	0.15%	70%
14 days	50%	0.25%	50%
30 days	75%	0.38%	25%
60 days	95%	0.48%	5%
90 days	100%	0.5%	0%

Payments under two weeks old have most of their fraud still hidden.

Training on the last 30 days of payments as if their labels were final. Most fraud in that window has not been reported yet, so it enters training as 'not fraud'.

Practise it: Design a Real-Time Fraud Detection System

Implicit feedback and its ambiguity

What it is

Implicit feedback is behaviour you read as a judgement. It is plentiful but each signal is ambiguous:

a click may mean interest, curiosity or a mis-tap;
no click may mean dislike, or that the user never scrolled that far;
time spent may mean engagement, or confusion;
a purchase is strong, but rare, and influenced by price and stock.

Why it matters

Most ranking and recommendation systems are trained on implicit feedback, so its biases become the model's.

Combining signals (click, add to basket, purchase, return) into a weighted label is common.
Negative signals (hiding an item, quick back-button) are valuable and often neglected.

In the real world: Video platforms moved from optimising clicks to optimising watch time and later satisfaction surveys, because clicks rewarded misleading thumbnails.

Treating every item not clicked as disliked. Items far down the screen were often never seen; counting them as negatives teaches the model that lower positions are bad items (module 7 covers position bias).

Human annotation

What it is

When labels must come from people, quality comes from process:

Guidelines: written definitions with examples of hard cases.
Training and a qualification test for annotators.
Gold questions: examples with known answers, mixed in to check each annotator.
Overlap: several annotators per example, with disagreements resolved by an expert.
Feedback: annotators see where they disagreed with the expert.

Why it matters

Labels from people are only as good as the instructions.

Vague guidelines produce labels that disagree, and a model cannot learn a pattern people themselves cannot agree on.
The hard, ambiguous cases are where the model most needs help, and where annotators most disagree.

In the real world: Content-moderation teams at large platforms keep guidelines of dozens of pages, test new reviewers on gold sets, and route disagreements to senior reviewers, whose decisions become the training labels.

Paying for the cheapest, fastest labels and never measuring agreement. The model's ceiling becomes the annotators' accuracy, and nobody knows what it is.

Measuring agreement between labellers

What it is

Agreement shows how reliable labels are. Two common measures:

Percent agreement: the share of examples two labellers marked the same.
Cohen's kappa: agreement beyond what chance would give. 1 is perfect, 0 is no better than chance; above about 0.6 is usually read as substantial.

Why it matters

Percent agreement flatters easy tasks.

If 95% of cases are clearly legitimate, two people who both say 'legitimate' every time agree 90% of the time without any skill.
Kappa corrects for that, so it shows whether the guideline really works.
Low agreement on a category tells you to rewrite its guideline, not to train harder.

In the real world: A team building a toxic-comment classifier measures kappa per category and finds 'sarcasm' at 0.3; it drops the category until the guideline can be made clearer.

Visual example

Computed

Two fraud reviewers, 100 cases. They agree 80% of the time, but chance alone would give 50%: kappa = 0.6.

Counts of each pair of decisions

	B: fraud	B: legitimate
A: fraud	40	10
A: legitimate	10	40

Cohen's kappa

po: observed agreement = (40 + 40) / 100 = 0.8

pe: chance agreement = 0.5×0.5 + 0.5×0.5 = 0.5

k: kappa = (0.8 − 0.5) / (1 − 0.5) = 0.6

Comparing a model's accuracy with a perfect 100% when people themselves agree only 80% of the time. The model cannot be judged more right than the labels it is checked against.

Noisy labels

What it is

Some labels are simply wrong. Causes include:

annotator mistakes and disagreement;
implicit signals misread (an accidental click);
delayed labels not yet final;
the labelling rule itself being imperfect.

Ways to cope: clean a trusted subset, down-weight doubtful examples, use losses that tolerate noise, and find suspect labels where a good model confidently disagrees with them.

Why it matters

A little noise is tolerable; systematic noise is not.

Random errors average out with enough data.
Errors that follow a pattern (one annotator always says 'fraud' for one country) teach the model that pattern.
A small, carefully labelled test set is essential even if training labels are noisy, or you cannot measure anything.

In the real world: A team reviews the 1,000 training examples where its model most confidently disagrees with the label, finds a third are mislabelled, fixes them, and gains more accuracy than a month of model tuning had.

Evaluating on labels as noisy as the training labels. The score then measures agreement with the noise, and a model that is genuinely better can look worse.

Weak supervision

What it is

Weak supervision creates labels from rules instead of people:

experts write several simple, imperfect labelling functions: 'if the email domain was registered yesterday, probably fraud';
each function labels some examples, sometimes wrongly, sometimes in conflict;
a model combines their votes, learning how accurate each function is, into one probabilistic label per example;
the final model is trained on those labels.

The best-known tool is Snorkel.

Why it matters

It turns expert knowledge into thousands of labels quickly.

Writing ten rules takes a day; labelling ten thousand examples takes weeks.
The final model generalises beyond the rules, because it learns from all the features, not just the ones the rules mention.

In the real world: Google described using weak supervision to build content classifiers from organisational knowledge in days rather than the months hand-labelling would have taken.

Expecting weak labels to replace a clean test set. You still need a small set labelled carefully by people to know whether the result works.

Active learning

What it is

Active learning chooses which examples people should label next, so each label teaches the model as much as possible:

train a model on the labels you have;
score the unlabelled pool;
send the examples the model is least sure about (scores near the threshold), or that are most unlike anything labelled, to annotators;
retrain and repeat.

Why it matters

Labelling budget is usually the constraint.

Random examples are mostly easy cases the model already gets right.
Uncertain cases sit near the decision boundary, where a label changes the model most.
It can reach the same accuracy with a fraction of the labels.

In the real world: Kirana Kart's fraud reviewers spend part of each day on payments the model scored between 0.4 and 0.6, chosen by active learning, instead of random ones.

Visual example

Loop

Each round spends the labelling budget where the model is least sure.

One round of active learning

TrainOn all labels so far
Score the poolPredict unlabelled examples
PickMost uncertain or unusual
LabelPeople label only those

Repeat until accuracy stops improving or the budget runs out.

Using only actively chosen examples to measure accuracy. They are deliberately the hardest cases, so the score will look worse than reality; keep a separate random sample for evaluation.

Practise it: Design a Large-Scale Classification Platform, Improve a Low-Recall Model

Labels shaped by the system itself

What it is

Sometimes you only see a label for cases your own system allowed:

a lender only learns whether approved customers repay; refused ones never get the chance;
a fraud system only learns about unblocked payments; blocked ones never become chargebacks;
a recommender only learns about items it showed.

These are selective labels: the data is shaped by past decisions.

Why it matters

A model trained only on what was allowed learns a distorted world.

It never sees the outcome of the cases it refused, so it cannot learn whether refusing them was right.
Over time it can lock in its own past mistakes (a feedback loop, module 22).
The standard fix is a small random holdout: a few cases let through or shown regardless, to keep learning the truth.

In the real world: Payment companies commonly let a very small random share of moderately risky payments through, or send them to review, specifically to keep getting unbiased labels.

Reading the model's falling fraud rate as success. If it blocks more and more, fewer fraud labels appear at all, simply because blocked fraud is never recorded.

Interview questions: labels

What it is

The questions interviewers ask about labels, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Fraud, credit and moderation interviews nearly always reach labels within ten minutes. 'How do you know it's fraud, and when?' is a favourite question.

Q: Where can labels come from?

Natural outcomes (the actual delivery time), implicit feedback (clicks, purchases), explicit feedback (ratings, reports) and human annotation. Each trades volume, accuracy, cost and delay.

Q: What is label delay and how do you handle it?

The time until the right answer is known, such as months for chargebacks. Train only on examples whose labels are final, use faster proxy signals for monitoring, and account for incomplete labels on recent data.

Q: How do you get good human labels?

Clear guidelines with hard examples, qualification tests, gold questions, overlapping annotators, expert adjudication and measured agreement such as Cohen's kappa.

Q: What is weak supervision?

Writing several imperfect labelling rules and combining their votes into probabilistic labels, then training a model on them. Fast, but it still needs a carefully labelled test set.

Q: What is active learning?

Choosing which examples to label next, typically the ones the model is least sure about, so each label improves the model most. Keep a random sample for honest evaluation.

Q: What are selective labels?

Labels you only observe for cases your system allowed, like repayment for approved loans. They bias training, so keep a small random holdout to observe outcomes for refused cases.

Saying 'we'll use historical data' without saying what the label is or when it arrives.

Practise it: Design Model Monitoring, Design a Search Ranking System

Exercise: define and audit a label

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: Teams that run a labelling pilot like this before scaling up routinely save weeks of relabelling later.

Your task. Write a precise label definition for a model, then measure how reliable and how delayed it is.

Steps

Choose a target: churn, late delivery, or fraud.
Write the definition: the event, the window, the start point, and the treatment of three edge cases.
Write the SQL that produces it, including the filter that excludes rows whose window has not closed.
Ask two friends to label the same 30 examples by your written guideline (use any public data, such as product reviews labelled helpful or not).
Compute percent agreement and Cohen's kappa.
Rewrite the guideline where you disagreed most and repeat on 30 new examples.

You are done when

Your SQL never labels an example whose window is still open.
You have kappa before and after rewriting the guideline.
You can name the edge case that caused most disagreement.

Stretch. Simulate label delay: hide labels for the last 20% of rows by date, train, and compare the model with one trained on full labels.