Design: Video Search

Clipstream, a made-up video platform, wants better search. Clarify:

• catalogue: about 500,000,000 videos; 20,000 hours uploaded a day;
• queries: short typed text ('how to fix a leaking tap', 'goal highlights last night', song names), many languages;
• latency: results within 200 ms;
• freshness: a new upload should be searchable within minutes; news and sport within seconds of being processed;
• goal: people find and watch what they searched for; satisfied searches, not just clicks.

Video is the hardest content to search.

• The words people type often appear nowhere in the title.
• What a video is about is in its speech, its pictures and its sound.
• So most of the work is turning video into searchable signals before anyone searches.

Video search is retrieval followed by ranking:

• retrieve candidates from several indexes: keyword search over text signals, and semantic search in a shared text-video embedding space;
• rank a few hundred candidates with a learning-to-rank model (module 14);
• labels: graded from behaviour after a search: watched to a meaningful length > clicked and left quickly > not clicked, plus human relevance judgements;
• offline metrics: NDCG at 10 on judged queries; recall of each retrieval path;
• online metrics: searches followed by a satisfying watch; quick returns to search (a sign of failure); query reformulation.

Watch time after a search is a better label than a click, because misleading thumbnails attract clicks that end in seconds.

• 'Satisfied search' (a watch of meaningful length) is the core online metric.

Each upload is processed into several signals:

• text: title, description, tags; speech transcript from automatic speech recognition (ASR); on-screen text from OCR;
• visual: embeddings of sampled frames (for example one frame a second), pooled into a video-level vector, and possibly per-segment vectors;
• audio: music identification; sound-event tags (applause, engine);
• derived: language, topic categories, detected faces of known public figures if policy allows, quality and safety scores.

Each signal catches queries the others miss.

• Speech transcripts answer 'how to' queries.
• Visual embeddings answer 'red sports car drifting'.
• Music identification answers song names.
• Combining them is what makes video findable regardless of its title.

Semantic retrieval needs queries and videos in the same vector space, so a typed query lands near videos that show or say what it describes:

• start from a pretrained image-text model trained on image-caption pairs (CLIP-style contrastive training);
• extend to video by pooling frame embeddings, or with a model trained on video-text pairs;
• fine-tune on Clipstream's own pairs: (query, video watched satisfyingly after that query);
• a text tower embeds queries at search time; video vectors are computed at upload and indexed (a two-tower design, module 13).

This is what lets 'puppy learning to swim' find a video titled 'Bruno's first day at the lake'.

• Fine-tuning on the platform's own search logs adapts general knowledge to how its users phrase things.

Run retrieval paths in parallel and merge:

• keyword (BM25) over title, description, transcript and on-screen text: precise for names, exact phrases, song titles;
• semantic over the text-video embedding space: descriptive and visual queries;
• fresh index: recent uploads in a small, frequently updated index (module 13);
• merge with reciprocal rank fusion: each list gives an item 1 ÷ (60 + its rank), summed across lists.

Fusion rewards items that appear high in more than one list without needing their scores to be comparable.

• BM25 scores and cosine similarities are on different scales, so adding them directly would let one dominate.

A learning-to-rank model over a few hundred candidates, with features in groups:

• relevance: keyword match scores per field, embedding similarity, transcript match, language match;
• quality: past satisfied-watch rate for this query or similar queries, overall completion rate, video and audio quality, channel reputation;
• freshness: upload age, and whether the query is news-like (detected by a sudden spike in its volume);
• personal: languages the user watches in, channels they follow, past watches on the topic;
• safety: policy and safety scores, used both as features and as hard filters.

Relevance decides which videos qualify; quality and freshness decide their order.

• For news-like queries, freshness should dominate: last night's highlights, not last year's.

Making uploads searchable is a streaming pipeline:

• upload event on the stream;
• fast path (seconds to minutes): title and description indexed; thumbnail and a few frames embedded; safety checks;
• full path (minutes): speech transcript, OCR, all sampled frames, audio identification, segment vectors;
• index updates: the fresh index first, merged into the main index on its rebuild schedule;
• re-processing of the whole catalogue whenever an embedding model changes, with an atomic index swap.

A two-speed pipeline gets new videos findable quickly while expensive analysis completes.

• Model upgrades are the expensive event: re-embedding hundreds of millions of videos must be planned as a large batch job.

Within 200 ms:

• query understanding (spelling, language, intent such as fresh or music) in a few milliseconds;
• retrieval paths in parallel, each with its own timeout;
• ranking in one batched call;
• cache results for popular queries for a short time, keyed by language and safety settings;
• fallback: keyword results alone if semantic retrieval is slow.

Parallel retrieval keeps the slowest path, not the sum, on the critical path.

• Popular queries repeat heavily, especially for news and music, so caching removes a large share of load.

Measure search from several angles:

• offline: NDCG at 10 on human-judged queries, by query type (navigational, how-to, music, news, visual description) and language;
• online: A/B tests on satisfied-search rate, quick-return rate and reformulation;
• monitoring: zero-result rate, latency by path, freshness lag (upload to searchable), share of queries answered mostly by one path, safety incidents in results.

Query types behave differently: a change can help descriptive queries and hurt song searches.

• Freshness lag is a key operational metric for news and sport.

Public work this design draws on:

• Radford et al., 'Learning Transferable Visual Models From Natural Language Supervision' (CLIP), 2021: contrastive image-text training;
• Radford et al., 'Robust Speech Recognition via Large-Scale Weak Supervision' (Whisper), 2022: multilingual speech transcripts;
• Miech et al., 'HowTo100M', 2019, and later text-video retrieval work: learning from narrated videos;
• Cormack et al., 'Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods', SIGIR 2009.

These papers explain the building blocks used by modern video search, and give concrete results to cite.

The questions interviewers ask about the video search design, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.