Ranking and Multi-Stage Systems

Large ranking systems pass items through several stages, each more accurate and more expensive per item than the last:

• Retrieval: thousands of candidates from the whole catalogue, in milliseconds.
• Light ranker (pre-ranking): a small, fast model cuts thousands to hundreds.
• Heavy ranker: the most accurate model scores the hundreds.
• Re-ranking: rules and adjustments (variety, freshness, business limits) produce the final list.

Each stage spends effort only where it matters.

• The expensive model sees only items that are already plausible.
• The total time fits the budget even though the heavy model alone could never score the whole catalogue.
• Stages can be improved independently, each with its own measure.

Three ways to train a ranking model, by what one training example is:

• Pointwise: one item. Predict its score (will it be added?) and sort by it. Simple; most common.
• Pairwise: two items. Learn which of the pair should be higher. Closer to what ranking needs. Example: RankNet, LambdaRank.
• Listwise: a whole list. Optimise a list measure such as NDCG directly. Example: LambdaMART, listwise softmax losses.

Pointwise is often good enough and gives calibrated probabilities that can be combined with other goals.

• Pairwise and listwise focus training on getting the order right at the top, which can improve the measures that matter for search.
• LightGBM and XGBoost offer LambdaMART-style ranking objectives with one setting.

Rankers combine four groups of features:

• User: history, preferences, segment, embedding.
• Item: category, price, popularity, quality, embedding.
• User-item: how this user relates to this item. Bought it before? Similarity of embeddings? Bought from this brand?
• Context: time, day, device, location, what is in the basket, the search query.

The user-item features are usually the most powerful, and they are what the ranker can do that two-tower retrieval cannot.

• 'This user bought this exact item four times' beats any general signal.
• Context changes what is relevant: breakfast items in the morning, party snacks on Friday evening.

Ranking usually serves several goals at once: engagement, revenue, long-term satisfaction. Common approach:

• predict each goal separately (or with a multi-task model): probability of adding, margin, chance of returning;
• combine them into one score with weights the business sets;
• test different weights with online experiments.

Optimising one goal alone hurts others.

• Pure engagement fills the screen with cheap, low-margin items.
• Pure revenue shows expensive items people do not want, and they stop opening the app.
• Weights make the trade-off visible and adjustable without retraining.

The heavy ranker scores items one at a time. The final list also needs list-level properties, added in re-ranking:

• variety (diversity): not ten kinds of milk in a row. A method such as MMR picks each next item for a high score and low similarity to items already chosen;
• freshness: some room for new items;
• business rules: no out-of-stock items, a cap on promoted items, legal restrictions;
• de-duplication: one listing per product.

A list of individually best items is often a poor list.

• Ten near-identical items waste the screen.
• Users engage more with varied lists, and the system learns more from them.
• Rules belong here, in readable code, not hidden in the model.

A ranker trained only on what it chose to show keeps showing the same things. Exploration deliberately shows some items it is less sure of, to learn about them:

• epsilon-greedy: a small share of the time, put a random candidate in a slot;
• Thompson sampling and upper confidence bound: favour items whose score is uncertain, because they might be better than thought;
• reserved slots for new items.

Without exploration the system learns only about what it already believes.

• New and niche items never get enough exposure to prove themselves.
• The data becomes narrower over time (a feedback loop, module 22).
• Exploration costs a little engagement now for better models later; the cost is measured and capped.

What the user is doing right now is often the strongest signal:

• items viewed or added in this session;
• the current search query and its refinements;
• what was just skipped or removed from the basket.

These are on-demand or streaming features (modules 9 and 10), or a sequence model over the session's events.

Intent changes within minutes.

• Someone who just added pasta wants sauce and cheese now, whatever they usually buy.
• A ranker using only yesterday's features cannot react.

To score hundreds of candidates inside the budget:

• fetch user and context features once, item features for all candidates in one batched read;
• score all candidates in one batched call to the model;
• cache item features and item-only parts of the model;
• keep a fallback order (popularity, or the retrieval order) if the ranker times out.

Batching is the difference between 40 ms and 400 ms.

• 500 separate model calls each pay the fixed overhead; one call with 500 rows pays it once.
• A timeout with a sensible fallback keeps the screen working when the ranker is slow.

The questions interviewers ask about ranking, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.