ML System Design · Module 34 of 47 · 12 concepts · about 15 min

Design: Blurring Faces and Number Plates in Street Imagery

A complete design for automatically finding and blurring every face and vehicle number plate in millions of street-level photos before they are published: object detection, why recall matters more than precision, measuring detection with overlap, removing duplicate boxes, a batch pipeline at scale, and human review for what the model misses.

At work: Mapping, real-estate, insurance and autonomous-driving companies all publish or store imagery of public streets and must protect bystanders' privacy. It is also the standard interview question for object detection.

Requirements

What it is

MapLane, a made-up mapping service, drives camera cars through cities and publishes street-level panoramas. Clarify:

volume: about 20,000,000 images a day from the fleet;
objects: human faces (including side views, small and distant ones) and vehicle number plates in many formats;
timing: nothing is published until it is processed, but there is no live user waiting: hours are acceptable;
quality bar: missing a face is a privacy failure; blurring a shop sign by mistake is a small cosmetic cost;
law: privacy laws (India's DPDP Act, Europe's GDPR) and the right to request extra blurring of a house or person.

Why it matters

The costs of the two mistakes are very unequal, and that one fact shapes the whole design.

A false negative (an unblurred face) is published to the world.
A false positive (a blurred statue) annoys almost nobody.
So the system is tuned for very high recall, accepting many extra blurs.

In the real world: Google's 2009 paper on Street View privacy described exactly this trade: the system was tuned for high recall and accepted false positives as the price.

Optimising a balanced measure such as F1. It treats a missed face and an extra blur as equally bad, which they are not here.

Framing as object detection

What it is

This is object detection: for each image, output a list of boxes, each with a class (face or plate) and a confidence score. Then:

every box above a low threshold is blurred;
the blur is irreversible in the published image (strong blur or pixelation, applied to the stored copy that is served);
offline metrics: recall at a fixed overlap, and average precision (the area under the precision-recall curve for detections);
online metrics: user reports of unblurred faces or plates per million images; reviewer-found misses in audits.

Why it matters

Detection, not classification, because one image contains many objects at different places and sizes.

Segmentation (pixel masks) is possible but unnecessary: a box-shaped blur is enough for privacy and much cheaper.

In the real world: Detection models such as Faster R-CNN, RetinaNet and the YOLO family are the usual starting points; most teams fine-tune a pretrained detector rather than train from scratch.

Classifying whole images as 'contains a face'. The system needs to know where every face is, not whether there is one.

Measuring a detection: overlap (IoU)

What it is

A predicted box counts as finding an object if it overlaps the true box enough. Overlap is measured by intersection over union (IoU):

the area both boxes share, divided by the area either box covers;
1 means identical boxes; 0 means no overlap;
a common rule: IoU of at least 0.5 counts as a match.

Recall and precision are then computed over matched boxes, and average precision summarises them across all confidence thresholds.

Why it matters

IoU turns 'roughly in the right place' into a number.

For privacy, a box that covers the face fully but is too large is fine; a box that covers half the face is not. Choose the matching rule with that in mind, and check coverage of the true box as well as IoU.

In the real world: Detection benchmarks such as COCO report average precision at several IoU thresholds (0.5 up to 0.95) for exactly this reason.

Visual example

Computed

True face box [100, 100, 200, 200]. Only some predictions overlap enough to count.

IoU = shared area ÷ combined area; match if ≥ 0.5

prediction	box	IoU	match?	share of the face covered
Almost exact	[105, 102, 205, 198]	0.87	Yes	91%
Shifted by a third	[133, 100, 233, 200]	0.5	Yes	67%
Too large	[80, 80, 240, 240]	0.39	No	100%
Half off	[150, 150, 250, 250]	0.14	No	25%

The oversized box fails the IoU rule yet covers the whole face: perfectly good for privacy.

Accepting a low-IoU match as a success for blurring. A box that overlaps only half a face leaves the other half recognisable.

Data and labels

What it is

Training data:

public datasets to start: face detection sets (such as WIDER FACE) and licence-plate sets;
MapLane's own images, labelled with boxes by annotators, because street panoramas have unusual distortion, angles and distances;
hard examples deliberately collected: tiny distant faces, faces behind glass, side views, plates at steep angles, night images, posters and statues (to learn what is *not* a real face, though blurring them is harmless);
reports from users of missed faces, which become new labels.

Why it matters

Public datasets differ from fisheye street panoramas: models trained only on them miss many small and distorted faces.

Label guidelines must say what to do with partial faces, crowds and reflections.
Box labelling is slower than image labelling, so active learning (module 6) is used to pick images worth labelling.

In the real world: Teams typically find that most misses are small, distant or partly hidden faces, and spend most of their labelling budget there.

Labelling only clear, frontal faces. The model then learns that small or turned faces are background, exactly the cases that matter.

Practise it: Design Model Monitoring, Design a Search Ranking System

The model

What it is

A practical design:

a pretrained detector, fine-tuned on street imagery, with two classes (face, plate);
multiple scales: run on the panorama at several resolutions, or in overlapping tiles, so tiny faces are large enough to detect;
a low score threshold for blurring, chosen to reach the recall target;
optionally a second, lighter verifier for borderline boxes, or a specialist plate reader for plate formats.

Why it matters

Tiling and multiple scales matter more than the detector architecture.

Panoramas are very large; shrinking them to the detector's input size makes distant faces a few pixels wide.
A low threshold buys recall; the extra blurs are acceptable by design.

In the real world: The 2009 Street View system combined several detectors with a post-processing classifier to raise recall, because a single off-the-shelf face detector was not enough.

def detect_panorama(image, detector, tile=1024, overlap=128, threshold=0.2):
    """Run the detector on overlapping tiles so small faces stay large enough to find."""
    boxes = []
    for x0, y0, crop in tiles(image, size=tile, overlap=overlap):
        for b in detector(crop):                       # boxes in tile coordinates
            if b.score >= threshold:
                boxes.append(b.shift(x0, y0))          # back to panorama coordinates
    return non_max_suppression(boxes, iou_threshold=0.5)

Tiles with no overlap. A face that falls on a tile edge is cut in two and may be missed in both halves.

Removing duplicate boxes

What it is

Detectors produce several overlapping boxes for one object, especially with tiles and multiple scales. Non-maximum suppression (NMS) removes duplicates:

sort boxes by confidence, highest first;
keep the top box; drop any remaining box that overlaps it by more than a set IoU;
repeat with the next box still standing.

Why it matters

Duplicates inflate counts and confuse evaluation (one face counted as three detections, two of them 'false positives').

For blurring, duplicates are harmless, but NMS still matters for honest measurement and for reviewers.

In the real world: Most detection libraries apply NMS by default; crowded scenes are where its threshold most needs tuning.

Visual example

Computed

Four detections, NMS at IoU 0.5. Two boxes remain: one per face.

Processed in order of confidence

box	confidence	coordinates	result
A	0.95	[100, 100, 200, 200]	kept
B	0.9	[110, 105, 210, 205]	dropped: overlaps A (IoU 0.75)
C	0.8	[300, 120, 380, 200]	kept
D	0.6	[95, 98, 190, 195]	dropped: overlaps A (IoU 0.8)

An NMS threshold so low that two people standing close together are merged into one box, leaving one face partly unblurred. For privacy, err towards keeping boxes, or blur the union of overlapping boxes.

The pipeline at scale

What it is

A batch pipeline, run as images arrive from the fleet:

ingest raw panoramas into restricted storage (only the pipeline can read them);
detect in parallel on GPU workers, by tile and scale;
blur every box above the threshold in the copy to be published;
audit sample to human reviewers;
publish blurred images; keep or delete raw images according to the retention policy.

Why it matters

No user is waiting, so throughput and cost matter, not per-image latency.

A queue of images with autoscaled GPU workers (scaling on queue depth, module 19) handles uneven fleet uploads.
Restricting raw images is part of the privacy design, not an afterthought.

In the real world: Large imagery pipelines run detection as a batch job on spot GPUs with checkpointing, since a lost worker only means reprocessing a few images.

Visual example

Computed

20,000,000 images a day at 40 ms of GPU time each (all tiles and scales).

Illustrative GPU time per image

input	value
GPU-hours a day	222
GPUs if spread evenly over 24 hours	10
GPUs to finish in an 8-hour night window	28

Publishing first and blurring later. Even a short window of unblurred images can be scraped and copied.

Human review and user reports

What it is

No detector reaches perfect recall, so people close the gap:

a random audit sample of processed images, reviewed for missed faces and plates, gives an honest recall estimate (prevalence sampling, module 32);
a report button on every published image lets anyone request blurring of a face, plate, house or vehicle;
reports are acted on quickly, and the missed cases become training labels;
for very sensitive places, extra review before publishing.

Why it matters

Audits measure what the model misses; reports fix what audits miss.

Together they provide a continuous supply of the hardest examples for retraining.

In the real world: Street-level imagery services publicly offer a 'report a problem' route for requesting additional blurring of faces, plates and homes.

Visual example

Pipeline

Raw images never leave restricted storage.

From camera to published panorama

Before anyone sees it
IngestRaw images to restricted storage
Tile and detectFine-tuned detector, several scales
Merge boxesNon-maximum suppression
BlurEvery box above a low threshold
After publishing
then, side by side
AuditReviewers check a random sample
Publish and listenReports become new labels

Measuring recall only on the labelled test set. Real-world misses cluster in conditions the test set under-represents (night, rain, crowds); audits on live output reveal them.

Evaluation and monitoring

What it is

Measure separately for faces and plates, and by condition:

offline: recall at the operating threshold and average precision, sliced by object size, lighting, weather, country (plate formats differ), camera model;
live: audit-estimated recall; reports per million images; blurred area per image (a sudden fall suggests the detector stopped working);
on every new camera or city: a labelled sample before full processing, since new hardware and new plate formats break models.

Why it matters

The failure that matters most (a missed face) is invisible in normal operation, so it has to be sampled for.

Slices by size and condition show where to collect more data.

In the real world: Teams launching in a new country commonly retrain or extend the plate detector, because plate shapes, colours and scripts differ.

Visual example

Slices

Report recall per slice, not only overall.

Where detectors usually struggle, and the remedy

slice	why it is hard	remedy
Tiny, distant faces	A few pixels after resizing	Tiling; higher resolution; more labels
Side and partial faces	Under-represented in public data	Targeted labelling; augmentation
Night and rain	Low contrast; reflections	Condition-specific data
New plate formats	Different shapes and scripts	Per-country data and evaluation
Panorama distortion	Fisheye stretching	Train on the camera's own images

Reporting one overall recall. A model at 95% overall may be at 70% for faces smaller than 20 pixels, which is where most of the remaining risk lies.

Practise it: Design a Large-Scale Classification Platform, Design a Predictive Maintenance System, Design a Real-Time Anomaly Detection System, Design a Recommendation System, Design Feature Monitoring, Design Model Monitoring, Diagnose Training-Serving Skew, Improve a Low-Recall Model, Design a Search Ranking System

Sources and further reading

What it is

Public work this design draws on:

Frome et al., 'Large-scale Privacy Protection in Google Street View', ICCV 2009: the original description of face and plate blurring at scale, tuned for recall;
Lin et al., 'Microsoft COCO: Common Objects in Context', 2014: the benchmark and IoU-based evaluation used across detection;
Ren et al., 'Faster R-CNN', 2015, and the YOLO family of detectors: common detector designs;
Yang et al., 'WIDER FACE: A Face Detection Benchmark', 2016: faces at many scales and poses.

Why it matters

Reading the original sources shows why each design choice exists, and gives you real numbers to quote in an interview.

In the real world: Interviewers are impressed when a candidate can name the public paper behind an industry system and say what it found.

Interview questions: the blurring design

What it is

The questions interviewers ask about the blurring design, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Detection questions in ML design interviews often take this form; the asymmetric cost of errors is the first thing interviewers want to hear.

Q: Why optimise recall over precision here?

A missed face is a privacy breach published to everyone; an extra blur on a sign costs almost nothing. So the threshold is low and the system accepts many false positives.

Q: Why detection rather than classification or segmentation?

We need every object's location to blur it; classification gives no location, and pixel masks are unnecessary because a box-shaped blur suffices.

Q: How do you detect very small faces in huge panoramas?

Run the detector on overlapping tiles and at several scales, so distant faces are large enough, then merge with non-maximum suppression.

Q: What is IoU and how is it used?

Intersection over union of a predicted and a true box; a match usually needs IoU of at least 0.5. For privacy, also check that the true box is fully covered.

Q: How do you know what the system misses?

Random audits of processed images by reviewers, user reports on published images, and per-slice recall by size, lighting and region.

Q: Is this a batch or online system?

Batch: images are processed before publishing, so throughput and cost matter, using queued GPU workers that scale with the backlog.

Optimising F1 or accuracy instead of recall at a fixed operating point.

Exercise: detect and blur faces

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A working, measured blurring pipeline is a strong computer-vision portfolio project with an obvious real-world purpose.

Your task. Build a small face-and-plate blurring pipeline and measure its recall honestly by object size.

Steps

Download a small public street-scene dataset with face or plate boxes (or label 100 of your own street photos).
Run a pretrained detector (for example a YOLO face model or OpenCV's DNN face detector) on whole images; record recall at IoU 0.5.
Add overlapping tiling and run again; merge with non-maximum suppression.
Lower the threshold until recall reaches 95%, and count the extra false positives.
Report recall for faces smaller and larger than 32 pixels.
Blur detected boxes with a strong Gaussian blur and check visually that blurred faces are unrecognisable.

You are done when

Tiling raises recall on small faces.
You can state the false positives you accepted to reach 95% recall.
Your report has recall by size, not only overall.

Stretch. Fine-tune the detector on 200 of your own labelled images and measure the change on small faces.