Design: Ad Click Prediction

Clarify:

• goal: ad revenue, while keeping the shopping experience good and giving advertisers fair results;
• placement: up to two sponsored slots in search results and one in the home-screen row;
• pricing: advertisers bid per click; a click is charged by a second-price rule (below);
• scale: every search and home view is an ad opportunity, millions a day, each with dozens to hundreds of eligible ads;
• latency: inside the search and home budgets (tens of milliseconds for ads);
• constraints: advertiser budgets must not be overspent; relevance minimums so ads match the query.

Ads have three parties: users, advertisers and the platform.

• Pure revenue maximisation shows irrelevant ads and drives users away.
• Advertisers leave if their money buys poor clicks.
• The design has to balance all three.

The core ML task is predicting the probability of a click (pCTR) for each eligible ad in each context. The auction then ranks ads by expected value:

• eCPM (expected revenue per thousand impressions) = bid × pCTR × 1,000;
• show the top ads by eCPM, above a relevance or quality minimum;
• second-price: a clicked ad pays just enough to have kept its position: the next ad's eCPM ÷ its own pCTR, never more than its bid.

Ranking by expected value rewards relevance: a cheaper ad that people click can beat an expensive one they ignore.

• The predicted probability enters the price directly, so its level matters, not only its order.

The data is enormous and very imbalanced:

• every ad impression logged with context, position and the auction's other candidates;
• clicks joined to impressions (seconds later);
• conversions (purchase of the advertised product) joined later, sometimes days later;
• click rates of a few percent or less, so 95–99% of rows are negatives.

Volume makes negative sampling standard (module 7), which then requires recalibration.

• Position strongly affects clicks; it must be logged and handled (module 7).
• Conversions are rarer and delayed but closer to what advertisers value; predicting them (pCVR) is a second, related task.

Ad features are mostly sparse and high-cardinality:

• IDs: user, ad, advertiser, product, query, store;
• context: query text and category, page, position, time, device;
• history: past click rates of this ad, this advertiser, this user on this category (with smoothing for rare ones);
• crosses: query × ad category, user segment × advertiser;
• embeddings of queries, products and users.

Handled with the hashing trick and learned embeddings (module 9).

Click behaviour depends on specific combinations: this kind of user, this query, this brand.

• Memorising frequent combinations (crosses, IDs) and generalising to rare ones (embeddings) is exactly what modern click models are built to do.

Click models have evolved through a well-known ladder:

• logistic regression on hashed sparse features and crosses: fast, calibrated, still a strong baseline;
• boosted trees feeding logistic regression (Facebook, 2014): trees discover feature combinations;
• deep CTR models: Wide & Deep, DeepFM, Deep & Cross Network: embeddings for sparse IDs plus layers that learn interactions automatically;
• multi-task models predicting click and conversion together, since conversion is rarer.

The data (billions of rows, huge ID spaces, valuable interactions) is where deep models clearly beat trees.

• Small gains in log loss translate directly into revenue at this scale.
• Simpler models remain useful as baselines and fallbacks.

Because the predicted probability sets rankings and prices:

• after negative sampling, correct the probabilities (module 7);
• then calibrate on recent, unsampled data (isotonic or Platt, module 12), often per segment (placement, device);
• monitor predicted ÷ actual click rate continuously; it should stay close to 1.

Miscalibration costs money on both sides.

• Too-high pCTR inflates eCPM and second prices: advertisers are overcharged relative to value and eventually cut spending.
• Too-low pCTR under-prices ads and loses revenue.
• Mixed calibration across segments (good on phones, bad on web) distorts which ads win.

Scale and freshness drive both:

• training: on very large data with sharded embedding tables (module 15); frequent incremental updates (several times a day) because ad inventory and click rates change fast; periodic full retrains;
• serving: retrieve eligible ads (targeting, budget left, relevance to the query), score them with the click model in one batch, run the auction, apply pacing;
• pacing: slow an advertiser's participation so the daily budget lasts all day rather than running out by 10 a.m.

New ads appear constantly and need predictions from their first impression.

• Incremental updates keep the model current; full retrains reset accumulated drift.
• Pacing protects advertisers from spending their budget on the morning's traffic alone, which improves their results and keeps them advertising.

Offline, then online with all three parties in view:

• offline: log loss (or normalised entropy, log loss relative to always predicting the average rate), calibration ratio, AUC; slices by placement, device, new ads;
• online: revenue per thousand opportunities, click-through, conversion rate for advertisers, cost per click;
• user guardrails: organic engagement, search success, complaints, ad 'hide' rate;
• advertiser guardrails: return on ad spend, budget delivery.

Revenue can rise at users' expense.

• Showing more and worse ads lifts revenue this week and erodes searches and orders over months.
• Long-term holdouts (users seeing fewer ads) measure that slow cost (module 17).

Watch money and quality in near real time:

• calibration ratio per placement, hourly;
• revenue, cost per click and click-through per hour against normal;
• budget pacing: campaigns over- or under-delivering;
• click fraud signals: bursts of clicks from one device or network (module 28);
• model freshness: time since last successful update.

Errors in ad systems cost money by the minute, so detection must be fast.

• A calibration break overcharges every advertiser until fixed, and refunds are painful.
• Click fraud inflates costs for advertisers and must be filtered from both billing and training data.

The questions interviewers ask about the ad click design, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.