ML System Design · Module 33 of 47 · 10 concepts · about 12 min

Design: Visual Search

A complete design for 'snap and shop': a customer photographs a product (an empty shampoo bottle, a packet in a friend's kitchen) and Kirana Kart finds it in the store. Image embeddings, fine-tuning on real customer photos, cropping, the index, re-ranking with text on the pack, and measuring recall on messy real photos.

At work: Asked as 'design image search', 'design visual product search' or 'find similar images'. It brings together embeddings, nearest-neighbour search and the gap between clean catalogue images and real-world photos.

Requirements

What it is

Clarify:

goal: the customer adds the product they photographed, in a few seconds;
inputs: phone photos: badly lit, angled, partly hidden, often of used or empty packs;
catalogue: 50,000 products per city with 4 clean studio images each; new products weekly; packaging redesigns;
latency: results within about a second end to end, including upload;
output: the exact product first if found; otherwise the closest alternatives in stock (same brand, similar product).

Why it matters

The core difficulty is the domain gap: catalogue photos are clean and front-on; customer photos are not.

A model trained only on catalogue images matches studio photos to studio photos and struggles with a crumpled pack on a kitchen counter.
'Exact product' versus 'similar product' are different tasks; the design must handle both.

In the real world: Pinterest Lens, Google Lens and Amazon's camera search all describe the gap between catalogue and real-world photos as their central problem.

Testing on catalogue images. Matching a catalogue photo to itself is easy; the real test is a phone photo.

Framing

What it is

Visual search is retrieval (module 13):

embed the query photo and every catalogue image into one vector space where the same product lands close together;
find nearest neighbours, restricted to the customer's store;
re-rank the top results with extra signals;
offline metrics: recall at 1 and at 10, and MRR, on real customer photos with known products;
online metrics: share of visual searches that end in an addition; time to addition.

Why it matters

Retrieval suits a large, changing catalogue.

A classifier with one class per product would need retraining for every new product; an index just adds vectors.
Recall at 1 matters most when the customer wants that exact product.

In the real world: Kirana Kart adds a new product to visual search the day it is listed, by embedding its studio images, with no retraining.

Framing it as classification over 50,000 products. Every new product or redesign then means retraining, and rare products with few examples are poorly learned.

Data

What it is

Training needs pairs of images of the same product, ideally across the domain gap:

catalogue images: several per product, from different angles;
customer photos linked to products: when a visual search ends in an addition, that photo and that product form a (noisy) positive pair;
labelled sets: a few thousand customer photos matched to products by annotators, for evaluation;
augmentation: blur, crop, rotate, change lighting and add background to catalogue images, to imitate phone photos.

Why it matters

Pairs across the domain gap are what teach the model that a crumpled real pack equals the studio image.

Behavioural pairs scale over time but contain mistakes (the customer added a different but acceptable product).
A clean, hand-matched evaluation set is essential; behavioural pairs are not trustworthy enough to judge with.

In the real world: Kirana Kart started with augmented catalogue images only; after three months of logging, it had 400,000 customer-photo pairs to fine-tune on, and recall at 1 rose sharply.

Using behavioural pairs for evaluation. They come from the current model's results, so they favour what it already finds and hide what it misses.

The model

What it is

Build on a pretrained vision model and adapt it:

start from a strong pretrained image encoder (a vision transformer or CNN, or an image-text model such as CLIP);
fine-tune with a contrastive loss: pull a customer photo towards its product's images and push it away from other products (in-batch negatives and hard negatives such as other sizes or flavours of the same brand);
optionally an image-text model, so a customer can also type 'like this but sugar free'.

Why it matters

Pretrained encoders already understand shapes, colours and text-like patterns; fine-tuning teaches what 'same product' means here.

Hard negatives matter most: the 500 ml and 1 litre bottles of the same shampoo look almost identical.

In the real world: Kirana Kart mines hard negatives as 'same brand, same category, different product', which reduced size and flavour mix-ups more than any other change.

import torch.nn.functional as F

def contrastive_loss(photo_emb, product_emb, temperature=0.05):
    """photo_emb[i] and product_emb[i] are the same product; all other pairs are negatives."""
    photo = F.normalize(photo_emb, dim=-1)
    product = F.normalize(product_emb, dim=-1)
    logits = photo @ product.T / temperature
    targets = torch.arange(len(photo))
    return (F.cross_entropy(logits, targets) + F.cross_entropy(logits.T, targets)) / 2

Fine-tuning only on easy negatives (random other products). The model learns to tell shampoo from rice, which it already could, and still confuses sizes and flavours.

Cropping the product

What it is

Customer photos contain more than the product: hands, counters, other items. Before embedding:

run an object detector to find product-like regions;
embed the largest or most central region, or several and search with each;
let the customer tap a different region if the first guess is wrong.

Why it matters

Background clutter pulls the embedding away from the product.

Cropping is one of the cheapest large improvements in visual search.
Embedding several regions also handles photos of a shelf with many products.

In the real world: Pinterest's visual search lets users adjust the crop box, and its papers report detection-plus-cropping as a major quality gain.

Embedding the whole photo. A packet on a patterned tablecloth then matches products with similar patterns.

Index, re-ranking and serving

What it is

Serve with a vector index and a re-ranking step:

index: all catalogue image vectors, one small index per store of what it stocks (module 13); ANN search (HNSW) in a few milliseconds;
re-rank the top 50: text read from the pack by OCR (brand, size, variant) compared with the product's details; stock; popularity;
serving: the phone resizes the photo before upload; detection and embedding on GPU; fallback to text search if no confident match.

Why it matters

OCR on the pack resolves what images cannot: '500 ml' versus '1 L', 'sugar free' versus regular.

A per-store index means every result is buyable.
A confidence threshold avoids showing a wrong 'exact match'; below it, show 'similar products' instead.

In the real world: Kirana Kart reads text from the pack and boosts candidates whose size and variant match; it fixed most of the remaining size errors.

Visual example

Computed

Server time for one visual search: 85 ms, well inside the one-second target including upload.

Index: 50,000 products × 4 images × 512 numbers at 16 bits = 204.8 MB per city

step	ms
Receive photo (already resized on the phone)	20
Detect and crop the product	25
Image embedding (GPU)	15
Nearest-neighbour search in the store's index	5
Re-rank top 50 (text on pack, size, stock)	20
Total	85

Always presenting the top result as the match. When confidence is low, saying 'similar products' is more honest and converts better than a confident wrong answer.

Practise it: Design a Recommendation System, Design Feature Monitoring, Diagnose Training-Serving Skew, Design a Model Serving Platform, Design a Personalized News Feed, Design a Search Ranking System, Design an Ad Ranking System, Design an ML Feature Store

New products and redesigns

What it is

The catalogue changes constantly:

new products: embed their studio images on listing and add them to the store indexes (incremental updates, module 13);
packaging redesigns: keep old and new images both linked to the product for a while, since customers photograph old packs at home;
model updates: re-embed every image with the new model and swap the whole index at once.

Why it matters

Customers photograph what they have at home, which may be last year's packaging.

Mixing embeddings from two model versions in one index breaks similarity (module 13).

In the real world: Kirana Kart keeps retired packaging images linked to their product for a year after a redesign.

Replacing a product's images on redesign. Photos of the old pack, which customers still own, no longer match anything.

Evaluation and monitoring

What it is

Measure on real photos and watch for gaps:

offline: recall at 1 and 10 and MRR on the hand-matched set of customer photos; slices by category, lighting, photo quality, packaging age;
online: A/B test with additions per visual search; share of searches with no confident match;
monitoring: no-match rate and confidence distribution; products frequently confused with each other (a confusion list for re-ranking rules or hard negatives); queries whose top result is out of stock.

Why it matters

Confusion pairs (products often mistaken for each other) are the most actionable output.

They become hard negatives for the next fine-tune and rules for re-ranking.
A rising no-match rate often means new products or redesigns not yet indexed.

In the real world: Kirana Kart excluded fresh produce from 'exact match' entirely after evaluation showed tomatoes of different varieties were indistinguishable from photos; it shows the category instead.

Visual example

Visual search

Crop, embed, search, re-rank, then decide how confident to be.

From a phone photo to results

PhotoResized on the phone
Detect and cropFind the product
EmbedFine-tuned image encoder
Searchtop 50Nearest in the store's index
Re-rankPack text: brand, size, variant
AnswerExact match, or similar products

A single overall recall number. Visual search can be excellent for packaged goods and poor for loose fruit and vegetables, which look alike to a model.

Practise it: Design a Large-Scale Classification Platform, Design a Predictive Maintenance System, Design a Real-Time Anomaly Detection System, Design a Recommendation System, Design Feature Monitoring, Design Model Monitoring, Diagnose Training-Serving Skew, Improve a Low-Recall Model, Design a Search Ranking System

Interview questions: the visual search design

What it is

The questions interviewers ask about the visual search design, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Visual search interviews usually probe the domain gap and how embeddings and indexes are kept consistent.

Q: Why retrieval rather than classification?

A catalogue changes weekly; retrieval adds new products by embedding their images, while a classifier needs retraining per product and learns rare products poorly.

Q: What is the domain gap, and how do you close it?

Catalogue images are clean; customer photos are messy. Augment catalogue images, fine-tune contrastively on real customer-photo pairs, and crop the product before embedding.

Q: How do you tell apart near-identical products?

Hard negatives (same brand, different size or flavour) in fine-tuning, and re-ranking with text read from the pack.

Q: How would you evaluate it?

Recall at 1 and 10 and MRR on a hand-matched set of real customer photos, sliced by category and photo conditions; online, additions per visual search and no-match rate.

Q: How do new products get in?

Embed their studio images on listing and add them to store indexes incrementally; after a model update, re-embed everything and swap the index at once.

Q: What if there is no good match?

Below a confidence threshold, show similar in-stock products or fall back to text search, rather than presenting a wrong exact match.

Training and testing only on catalogue images.

Exercise: build image retrieval

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A demo like this, measured on real photos, is a strong talking point in computer-vision interviews.

Your task. Build a small visual product search and measure it on photos taken with your own phone.

Steps

Collect 50 products' catalogue images from a public dataset or online shop pages (four each if possible).
Photograph 30 of those products yourself at home in varied conditions; record which product each photo shows.
Embed all images with a pretrained model (CLIP or a vision transformer) and index them with FAISS.
Measure recall at 1 and at 5 for your phone photos, with and without cropping the product first (by hand or with a detector).
Add a re-ranking step using OCR text from the photo matched against product names, and measure again.

You are done when

You have recall at 1 and 5 for each version.
Cropping and OCR re-ranking each measurably help.
You can name the product pairs most often confused.

Stretch. Fine-tune the encoder contrastively on augmented catalogue images and measure the change on your phone photos.