Bezen

ML System Design · Module 32 of 47 · 11 concepts · about 13 min

Design: Content Moderation

A complete design for checking what Kirana Kart's users post (product reviews with text and photos, and photos sent as evidence for refunds): turning policies into labels, a cascade from cheap rules to large models to people, review queues, reused-photo detection, evasion, and measuring how much bad content gets through.

At work: Asked as 'design a system to detect harmful content', 'design spam detection' or 'design image moderation'. It combines classification, human-in-the-loop operations, adversaries and policy.

Requirements

What it is

Clarify:

  • volume: about 80,000 items a day: reviews (text, often Hindi, English or both, some with photos) and refund-evidence photos;
  • policies: abuse and hate, spam and promotion, personal information (phone numbers, addresses), fake or paid reviews, and for refunds, photos that are reused, edited or of a different product;
  • timing: reviews checked before they are published, within minutes; refund photos within seconds, since an automatic refund depends on them;
  • actions: publish, remove, hold for review, or (for refunds) approve, deny or send to support;
  • people: a moderation team of 10; users can appeal.

Why it matters

Policies differ in harm and in difficulty, so they get different thresholds and different routes.

  • Personal information must almost never be published; mild spam is less urgent.
  • Refund photos are a fraud problem in disguise (module 28).

In the real world: Large platforms publish community guidelines that map one-to-one onto classifiers and review queues; changes to the policy text trigger relabelling.

Starting with 'a toxicity classifier'. Moderation is a set of policies, each needing a definition, labels and a decision rule.

From policy to labels

What it is

Every policy needs an operational definition before any model:

  • a written guideline with clear examples and hard cases (is a rude review of a product abuse, or honest criticism?);
  • labels per policy (multi-label: one review can be spam and contain a phone number);
  • severity levels where it matters (mild insult versus threat);
  • measured agreement between labellers, with guidelines revised where it is low (module 6).

Why it matters

The model can only be as consistent as the policy.

  • If moderators disagree a third of the time, no model will reach high precision, and evaluation numbers are unreliable.
  • Policy changes are frequent; a clean link from guideline version to label set makes relabelling manageable.

In the real world: Kirana Kart versions its review guidelines and records the version with every human decision; training uses only decisions made under the current version, plus re-labelled samples.

Training on whatever moderators did in the past, across several policy versions. The model learns a blur of old and new rules.

Models

What it is

Different content needs different models:

  • text: a fine-tuned multilingual transformer (small enough to run in milliseconds) for each policy, sharing one encoder with several heads (module 12);
  • images: embeddings from a pretrained vision model, with small classifiers on top for each policy;
  • pattern detectors: regular expressions and named-entity models for phone numbers and addresses;
  • large models: a multimodal model or a language model prompted with the policy, used only for the uncertain cases (the cascade below).

Why it matters

Fast, specialised models handle the bulk cheaply and consistently.

  • Large general models handle nuance (sarcasm, context, new slang) but are slow and expensive per item.
  • Shared encoders and embeddings mean adding a policy is training a small head, not a new model.

In the real world: Many trust-and-safety teams now use language models to pre-label and explain borderline cases for human reviewers, while fast classifiers make most decisions.

Sending everything to a large language model. At tens of thousands of items a day the cost and latency add up, and results vary with prompt wording; use it where it adds most.

Practise it: Design a Recommendation System, Design a Search Ranking System

A cascade from cheap to expensive

What it is

Route each item through stages, stopping as soon as one is confident:

  • rules and hashes: block known spam phrases, known bad images, obvious personal data;
  • fast classifier: publish clearly fine items, remove clearly violating ones;
  • large model: decide most of the uncertain middle, with a written reason;
  • human reviewers: the rest, plus appeals and a random sample for quality.

Why it matters

Most content is clearly fine; spending human time on it is waste.

  • Each stage handles what it does well and passes on only what it cannot decide.
  • Thresholds at each stage trade cost against the error that stage is allowed to make.

In the real world: Kirana Kart's ten moderators review about 2,000 items a day, about 2.5% of everything posted, because the earlier stages resolve the rest.

Visual example

Computed

80,000 items a day. People see 1,960 (2.5%); the whole cascade costs ₹17,326 a day against ₹6.4 lakh for reviewing everything by hand.

Illustrative resolve rates and costs per item
stageitems arrivingresolved herecost a day
Rules and known-bad hashes80,0001,600₹0
Fast classifier78,40070,560₹78.4
Large model (multimodal or LLM)7,8405,880₹1,568
Human reviewer1,9601,960₹15,680

Thresholds set so that the fast classifier decides almost everything. Its errors at the edges become the platform's errors; let it pass on more uncertain cases if review capacity allows.

Review queues

What it is

Human review is a system to design, not an afterthought:

  • prioritise by severity and reach (a threat before a mild insult; an item on a popular product page first);
  • show reviewers the model's reasons and the relevant policy section;
  • measure reviewer accuracy with gold items and a second review of a sample;
  • feed decisions back as labels, weighted by reviewer accuracy;
  • protect reviewers' wellbeing for disturbing content (blurring, limits, rotation).

Why it matters

Reviewers are both the last line of defence and the main source of high-quality labels.

  • A queue sorted by arrival time leaves the worst content waiting behind the mildest.
  • Their decisions improve the next model, so their quality is the model's ceiling.

In the real world: Kirana Kart routes 5% of each reviewer's decisions to a second reviewer; disagreements go to a lead and become guideline examples.

Ignoring reviewer disagreement. Two reviewers who would decide the same item differently produce contradictory labels that confuse the next model.

Reused and edited photos

What it is

Refund abuse often uses the same photo many times, or a photo from the internet. Detect it with perceptual hashing:

  • compute a short fingerprint of each image that changes little under resizing, cropping or recompression (pHash, PDQ);
  • compare new photos' hashes against past refund photos and against known stock images;
  • a near match (small Hamming distance between hashes) flags a reused photo;
  • embeddings catch looser matches (the same item photographed again from a different angle).

Why it matters

It catches a common abuse pattern exactly and cheaply.

  • Exact file hashes miss any re-saved or cropped copy; perceptual hashes do not.
  • Hash lookups are fast enough for the seconds-long refund decision.

In the real world: Meta open-sourced the PDQ perceptual hash for finding near-duplicate images at scale; similar techniques are used against reused images in refund and insurance claims.

import imagehash
from PIL import Image

def reused_photo(path: str, index, max_distance: int = 6):
    """Return earlier refund claims whose photo is a near-duplicate of this one."""
    h = imagehash.phash(Image.open(path))
    return [claim for claim, past in index.near(h) if h - past <= max_distance]   # Hamming distance

Matching only within one customer's history. Abuse rings share photos across many accounts; search all recent claims.

Evasion

What it is

People who want their content through will adapt:

  • misspellings, spacing and symbols ('c a l l 9 8 7...', 'fr33');
  • text inside images to get past text models;
  • mixing languages and scripts;
  • slightly edited images to change exact hashes.

Defences: normalisation (strip spacing, map look-alike characters), text extraction from images (OCR), character-level and multilingual models, perceptual rather than exact hashes, and fast retraining on newly caught examples.

Why it matters

Moderation is adversarial (module 25), so a static model decays quickly.

  • Reviewer decisions on newly evading content are the fastest source of labels for the next update.

In the real world: Kirana Kart runs OCR on every review photo and passes the extracted text through the same personal-information detector as the review itself.

Only checking the review text field. Phone numbers in a photo's caption, or written on the product in the photo, go straight through.

Practise it: Improve a Low-Recall Model

Measuring how much gets through

What it is

Precision is easy to measure (review what was removed). Recall is harder: what was missed is, by definition, not flagged. Measure it by:

  • prevalence sampling: regularly have reviewers label a random sample of published content and estimate the share that violates policy;
  • user reports as a signal (biased, but fast);
  • per-policy, per-language metrics.

Why it matters

Prevalence is the honest top-line number: how much bad content users actually see.

  • Without it, a model can look better simply by flagging less (fewer flags, fewer visible mistakes).
  • Rare policies need larger samples to measure.

In the real world: Large platforms publish prevalence estimates for categories of harmful content, measured by labelling random samples of viewed content.

Visual example

Measures

Recall needs random sampling of what was not flagged.

What to measure, and how
measurehowwhy
Precision per policyReview a sample of removalsAre we removing good content?
PrevalenceLabel a random sample of published itemsHow much bad content gets through
Appeal overturn rateShare of appeals upheldFairness of decisions
Time to actionMinutes from post to decisionExposure before removal
Reviewer agreementSecond reviews; gold itemsLabel quality

Reporting only 'items removed'. A rise could mean more abuse or better detection; a fall could mean less abuse or worse detection.

Monitoring and policy change

What it is

Watch the system and its inputs:

  • volume per policy and per stage; review queue length and age;
  • share of items reaching humans (a jump means the fast model is unsure: drift or a new pattern);
  • prevalence and appeals weekly; precision by language;
  • after a policy change, relabel a sample, retrain the affected heads and re-measure.

Why it matters

A rise in uncertain items is the earliest sign of a new pattern or an evasion technique.

  • Queue age protects users: a backlog means harmful items stay up longer, or reviews stay unpublished.

In the real world: When Kirana Kart allowed reviews mentioning competitor prices, it relabelled 3,000 past examples under the new rule and retrained the spam head within a week.

Visual example

Moderation

Most items stop at the first or second stage.

One review, from post to decision

  1. NormaliseClean text; OCR; image hashes
  2. Rules and hashes2% decidedKnown spam and bad images
  3. Fast models90% decidedPer-policy classifiers
  4. Large model75% decidedUncertain cases, with reasons
  5. Human review~2.5% of allThe rest, by severity
  6. Act and learnDecisions become labels

Changing the policy text without updating labels and thresholds. The model keeps enforcing the old rule, and reviewers enforce the new one, so the two disagree on every edge case.

Practise it: Design Feature Monitoring, Design Model Monitoring

Interview questions: the moderation design

What it is

The questions interviewers ask about the moderation design, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

  • Each answer below is the two or three sentences an interviewer is listening for.
  • The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Trust-and-safety interviews focus on the human loop and on measuring prevalence; the cascade diagram is expected.

Q: How would you structure a moderation system?

A cascade: rules and hashes, fast per-policy classifiers, a large model for uncertain cases, and human review for the rest, with thresholds at each stage set by cost, harm and review capacity.

Q: How do you get labels?

Versioned written guidelines with hard examples, trained reviewers, measured agreement, gold items, and reviewer decisions fed back as labels weighted by accuracy.

Q: How do you measure recall when misses are unflagged?

Prevalence sampling: label a random sample of published content to estimate how much violating content gets through, per policy and language.

Q: How do you detect reused images?

Perceptual hashes that survive resizing and recompression, compared across all recent submissions, plus embeddings for looser matches.

Q: How do people evade moderation, and what do you do?

Misspellings, symbols, text in images, mixed languages, edited images. Normalise text, OCR images, use multilingual character-aware models and perceptual hashes, and retrain quickly on new cases.

Q: Where do large language models fit?

On the uncertain middle and as assistants to reviewers, giving reasons; not on every item, because of cost, latency and variability.

A single classifier with a single threshold for every policy.

Exercise: build a two-stage moderator

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

  • Building it once, even in fifty lines, is what makes it yours.
  • Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: Trust-and-safety teams use this kind of experiment to argue for review headcount and model investment.

Your task. Build a cascade for one policy on public data and show how routing uncertain cases changes cost and quality.

Steps

  1. Use a public toxic-comment or spam dataset.
  2. Write normalisation and a few rules; measure what they catch alone.
  3. Train a fast classifier (TF-IDF with logistic regression, or a small transformer).
  4. Choose two thresholds so that only a set share of items (say 5%) is 'uncertain'.
  5. Send the uncertain items to a stand-in for a large model (a bigger model, or an LLM if you have access) and measure its accuracy on them.
  6. Compute precision, recall and cost per thousand items for: rules only, classifier only, and the full cascade with the remaining items counted as human-reviewed.

You are done when

  • The cascade beats the classifier alone on precision and recall.
  • You can state the cost per thousand items and the share sent to people.
  • You estimate recall on a random sample of items the system allowed.

Stretch. Add obfuscated examples (spaced letters, look-alike characters) and show how normalisation recovers recall.