ML System Design · Module 27 of 47 · 10 concepts · about 12 min

Design: Search Ranking

A complete design for Kirana Kart's product search: understanding messy, mixed-language queries, finding matching products with keyword and semantic retrieval together, ranking them with learning to rank, and measuring search quality with clicks and human judgements.

At work: Asked as 'design search for an e-commerce site', 'design a job search' or 'design web search ranking'. Search differs from recommendation in one way that changes everything: the user states what they want.

Requirements

What it is

Clarify:

goal: customers find and add what they searched for, quickly; fewer searches that end with nothing added;
scale: 3,000,000 searches a day, about 104 a second at peak; 50,000 products per city;
latency: results within 150 ms (p99), and suggestions as the user types within 50 ms;
queries: short, misspelt, mixed Hindi and English ('doodh', 'atta 5kg', 'amul buter');
constraints: only products in stock at the customer's store; exact brand searches must show that brand first.

Why it matters

Search has a stated intent, so relevance comes first and personalisation second.

A wrong result for 'Amul butter' is a failure no matter how well it suits the user's taste.
Mixed-language, misspelt queries make query understanding a major part of the design.

In the real world: Grocery search queries in India are often transliterated Hindi or regional words typed in English letters, so a purely English keyword index misses many of them.

Designing search like recommendations, leading with personalisation. Personalise only among relevant results.

Framing

What it is

Search is a ranking problem over (query, product) pairs:

target: graded relevance of a product to a query: 3 bought, 2 added, 1 clicked, 0 shown only (module 16), supplemented by human judgements;
task: retrieve a few hundred matching products, then rank with a learning-to-rank model (module 14);
offline metrics: NDCG at 10 on clicks and on a human-judged set; recall of retrieval;
online metrics: searches that lead to an addition; time to first addition; reformulation rate (user retypes the query); zero-result rate;
guardrails: latency, revenue per search.

Why it matters

Click-based labels are plentiful but biased by position and by what was shown.

Human judgements are unbiased but expensive; a few thousand queries judged by trained raters anchor offline evaluation.
Reformulation and zero-result rates are direct signs of failure that need no labels.

In the real world: Large search teams (web, e-commerce, jobs) maintain human-rated query sets precisely because click data cannot be trusted alone.

Using only click-through rate as the online metric. Showing tempting but irrelevant items raises clicks and frustrates people searching for something specific.

Query understanding

What it is

Before retrieving anything, interpret the query:

normalise: lower case, remove extra spaces, standardise units ('5 kg', '5kg');
correct spelling: 'buter' → 'butter', using a model trained on query logs and the catalogue's vocabulary;
transliterate and translate: 'doodh' → milk, 'atta' → wheat flour;
detect intent: a brand ('Amul'), a category ('vegetables'), an attribute ('organic', 'sugar free'), a quantity;
expand: add synonyms ('curd' ↔ 'dahi' ↔ 'yogurt').

Why it matters

Most search failures start with a misunderstood query.

A third or more of grocery queries may contain a misspelling, a transliteration or a size.
Detected brand and category become strong ranking features and filters.

In the real world: Kirana Kart learns many transliterations and synonyms from its own logs: queries that are followed, within the same session, by a reformulated query that leads to an addition.

def understand(raw: str) -> Query:
    q = normalise(raw)                         # "Amul  Buter 500G" -> "amul buter 500 g"
    q = spell.correct(q, vocab=catalogue_vocab) # -> "amul butter 500 g"
    q = transliterate(q)                        # "doodh" -> "milk" (both kept as terms)
    return Query(
        text=q,
        brand=brand_tagger(q),                  # "amul"
        category=category_classifier(q),        # "dairy > butter", with a confidence
        size=parse_size(q),                     # 500 g
        synonyms=synonyms.expand(q),            # "butter" -> also "makhan"
    )

Over-correcting. Changing a rare but valid brand name into a common word sends the customer to the wrong product; keep the original query too, and only correct with high confidence.

Practise it: Design a Search Ranking System

Retrieval: keyword and semantic together

What it is

Use two retrieval methods and merge them (hybrid retrieval):

lexical (keyword) search with BM25 on titles, brands and attributes, in Elasticsearch or OpenSearch: precise for exact words, brands and sizes;
semantic search with embeddings of queries and products (a two-tower model, module 13): finds products described differently ('healthy breakfast' → oats, muesli);
merge with reciprocal rank fusion or a learned blend;
filter to products in stock at the customer's store.

Why it matters

Each catches what the other misses.

Keyword search nails 'Amul butter 500 g' and fails on 'something for a sore throat'.
Semantic search handles vague and descriptive queries but can drift from exact brands and sizes.
Together they give high recall for the ranker.

In the real world: Many e-commerce search teams have described adding semantic retrieval beside keyword search, rather than replacing it, as the change that most reduced zero-result and poor-result searches.

Visual example

Two retrieval paths, one ranker.

From typed query to results

Understand the querySpelling, Hindi words, brand, size
two searches, in parallel
Keyword search~300Exact words: brands, sizes
Meaning search~300Similar products by embedding
Merge and filter~500Fuse; in stock at this store
RankLearning-to-rank model
Resultstop 20Exact brand first; one sponsored slot

Replacing keyword search with embeddings entirely. Exact matches on brands, sizes and product codes then get worse, and those are the queries customers are least forgiving about.

Practise it: Design a Recommendation System, Design an Image Search System, Design a Search Ranking System

Ranking features

What it is

A learning-to-rank model (LightGBM lambdarank, module 14) uses:

query-product match: BM25 score, embedding similarity, brand match, category match, size match;
product quality: popularity for this query (past add rate), overall sales, rating, stock depth;
personal: has this customer bought this product or brand; their usual pack size;
context: store, time of day, device.

Why it matters

Query-product match decides relevance; the rest decides order among relevant products.

'Past add rate for this query' is often the strongest single feature, learned from logs.
Personal features help most for broad queries ('milk': which brand, which size).

In the real world: For 'milk', Kirana Kart puts the customer's usual brand and size first; for 'Amul gold 1 litre' it shows exactly that, whatever the customer usually buys.

Letting popularity outweigh match. Then 'organic milk' returns the best-selling ordinary milk, because it sells most.

Practise it: Design a Recommendation System, Design a Personalized News Feed

Labels and their biases

What it is

Two sources of labels, used together:

behavioural: clicks, additions and purchases after each search, graded; corrected for position bias (module 7) using small randomised swaps of adjacent results;
human judgements: trained raters grade a sample of (query, product) pairs from 0 (irrelevant) to 3 (exact match), following written guidelines (module 6).

Why it matters

Behavioural labels scale; human labels are unbiased and cover cases the current ranker never shows.

Human labels also define relevance for rare and new queries with too little traffic.
Agreement between raters is measured, and guidelines refined, as in module 6.

In the real world: Kirana Kart has 5,000 queries judged by raters, refreshed monthly with new and failing queries; NDCG on this set is a release gate.

Training only on clicks from the current ranker. Products it never shows high up can never earn clicks, so the next model keeps ignoring them.

Practise it: Design Model Monitoring, Design a Search Ranking System

Serving

What it is

Meeting 150 ms at scale:

query understanding models are small and fast (under 10 ms);
the two retrieval calls run in parallel;
the ranker scores all candidates in one batched call;
cache results for the most popular queries per store for a few minutes (invalidated by stock changes);
typeahead suggestions served from a precomputed prefix index of popular queries.

Why it matters

Search queries are highly repetitive at the top: a cache removes a large share of the work.

Parallel retrieval keeps the slowest path, not the sum, on the critical path.

In the real world: About 40% of Kirana Kart searches are served from the per-store cache of popular queries.

Visual example

Computed

Peak 104 searches a second; 40% served from cache. The ranker sees 62 a second.

Illustrative cache hit rate for popular queries per store

	searches a second at peak
All searches	104
Served from cache	42
Reach retrieval and ranking	62

Caching results without the store in the key. Customers see products not stocked at their store.

Practise it: Design Feature Monitoring, Diagnose Training-Serving Skew, Design a Model Serving Platform, Design an ML Feature Store

Evaluation and monitoring

What it is

Measure search quality from several angles:

offline: NDCG at 10 on human judgements and on clicks; recall of each retrieval path; slices by query type (brand, category, misspelt, Hindi);
online: A/B test with searches-leading-to-addition as primary; reformulation and zero-result rates; interleaving for quick ranker comparisons;
monitoring: zero-result rate by store, top failing queries daily, latency, cache hit rate, stock-filter drop rate.

Why it matters

The daily list of top failing queries is the most actionable artefact in search.

It shows missing synonyms, new products, catalogue errors and misunderstood words, each easy to fix.
Slices by query type catch a change that helps brand searches and hurts vague ones.

In the real world: Kirana Kart reviews the 100 queries with the highest reformulation rate each week; most fixes are synonyms and catalogue corrections, not model changes.

Visual example

Signals

None needs labels.

Search health signals and what each reveals

signal	reveals
Zero-result rate	Missing products, synonyms or spellings
Reformulation rate	Results that did not match intent
Search exits with no addition	Overall search failure
Time to first addition	Good results buried too low
Stock-filter drop rate	Store stock problems hiding results

Looking only at average NDCG. A model can raise it on popular, easy queries while getting worse on the long tail where customers are most likely to give up.

Practise it: Design a Large-Scale Classification Platform, Design a Predictive Maintenance System, Design a Real-Time Anomaly Detection System, Design a Recommendation System, Design Feature Monitoring, Design Model Monitoring, Diagnose Training-Serving Skew, Improve a Low-Recall Model, Design a Search Ranking System

Interview questions: the search design

What it is

The questions interviewers ask about the search design, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Search interviews dig into query understanding and labels more than any other design; candidates who mention reformulation rate and human judgements stand out.

Q: How is search ranking different from recommendation?

The user states intent, so relevance to the query dominates and personalisation only orders relevant results. Query understanding and lexical matching become central.

Q: Why hybrid retrieval?

Keyword search is precise for exact brands, sizes and codes; semantic search handles vague or differently worded queries. Fusing them gives the ranker high recall.

Q: How do you get relevance labels?

Graded behavioural signals corrected for position bias, plus human-judged query-product pairs following written guidelines, which also cover rare and new queries.

Q: How do you handle misspellings and Hindi queries?

Spell correction trained on query logs and catalogue vocabulary, transliteration, and synonyms learned from reformulations, keeping the original query and correcting only with confidence.

Q: What would you monitor?

Zero-result and reformulation rates, searches ending without an addition, latency, cache hit rate, and a daily list of top failing queries.

Q: How would you personalise search?

Personal features in the ranker (past purchases of the product or brand, usual size) that matter most for broad queries, never overriding an exact match.

Proposing pure semantic search with no keyword matching.

Practise it: Design an Image Search System, Design a Search Ranking System

Exercise: build hybrid search and judge it

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A small judged query set like this is how real search teams begin before they have enough traffic for click labels.

Your task. Build keyword plus semantic retrieval over a product catalogue, rank with simple features, and evaluate on judgements you create.

Steps

Download a public product catalogue (for example an open grocery or e-commerce dataset).
Write 50 realistic queries, including misspellings, brands, sizes and vague ones; judge the top results for each from 0 to 3 yourself.
Index titles with BM25 (rank_bm25 or OpenSearch) and with sentence embeddings (FAISS).
Fuse the two lists with reciprocal rank fusion.
Measure NDCG at 10 for keyword only, semantic only and hybrid, overall and by query type.
Add a spell-correction step and measure again on the misspelt queries.

You are done when

Hybrid beats both single methods overall.
You can name a query type where keyword wins and one where semantic wins.
Spell correction measurably helps misspelt queries without hurting brand queries.

Stretch. Train a LightGBM lambdarank model on features from both retrievers and compare it with fusion.