RAG · Module 14 of 14 · 8 concepts · about 12 min

The RAG Interview

How to answer RAG questions in an interview: a structure for design questions, the numbers to estimate, the trade-offs to name, and the most frequently asked questions with the answer each one is looking for.

At work: Interviews for AI engineer, ML engineer and applied scientist roles, where 'design a system that answers questions over our documents' is now one of the most common design questions.

A framework for RAG system design questions

What it is

A design question ("design a support assistant over our help centre") is answered in a fixed order.

1. Clarify the requirements.
2. Estimate the scale.
3. Sketch the two pipelines, indexing and answering.
4. Go deep on the choices that matter: chunking, hybrid retrieval, reranking, prompt and grounding.
5. Evaluation: how you will know it works.
6. Production: access control, freshness, latency, cost, monitoring.
7. Failure modes and what you would do next.

Why it matters

Interviewers grade structure as much as content.

A candidate who starts by naming a vector database has skipped the two steps that decide whether a vector database is even needed.
Following the order shows you can run a project, and it guarantees you reach evaluation and access control, the two areas most often left out.

In the real world: A candidate who spends two minutes on requirements, draws two pipelines and reaches evaluation and permissions unprompted is rated above one who gives a deeper but unstructured tour of vector indexes.

Visual example

Interview

Roughly five minutes each. Do not skip the first two or the last three.

The order of a RAG design answer

ClarifyUsers, documents, freshness, permissions
EstimateChunks, memory, questions a second, cost
Two pipelinesIndexing and answering
Deep divesChunking, hybrid search, reranking
EvaluationGolden set, metrics, release gate
ProductionAccess, freshness, latency, cost
Failure modesWhat breaks first, and the fix

Spending the whole interview on retrieval. Leave time for evaluation and production; they are where senior candidates are told apart from junior ones. If time is short, say in one sentence each what you would do for them.

Clarifying requirements

What it is

The questions to ask before designing anything.

Who are the users, and how many?
What documents: how many, what formats, which languages, how often do they change?
What questions: lookups of single facts, comparisons, summaries, counting?
Permissions: may every user see every document?
Freshness: how soon must a change appear?
Quality bar: what does a wrong answer cost, and should the system refuse when unsure?
Speed and budget: a target response time and a cost per question?

Why it matters

Each answer changes the design.

PDFs with tables mean a layout-aware parser. Counting questions mean a database route.
Per-user permissions mean pre-filtered search.
A high cost of error means abstention and verification.
Minute-level freshness means incremental indexing from a change feed.
Asking shows that you know which facts drive which decisions.

In the real world: Told that 'employees can only see their own department's files', a strong candidate immediately adds permissions to chunk metadata and says the search must be pre-filtered.

Visual example

Requirements

Ask the question, then state the consequence.

Each requirement and the design decision it drives

ask about	it decides
Document formats (PDF, tables, scans)	Parser choice; table and OCR handling
Collection size and growth	Exact search or ANN; one machine or several
How often documents change	Nightly batch or incremental indexing
Who may see what	ACL metadata and pre-filtered search
Question types	Plain RAG, routing to a database, or multi-hop
Cost of a wrong answer	Abstention, citation verification, human handover
Response-time target	Which optional steps you can afford

Asking a long list of questions and then ignoring the answers. After each answer, say what it changes: 'documents change hourly, so I will index incrementally from change events instead of a nightly batch.'

Back-of-the-envelope numbers

What it is

Four quick estimates.

Chunks = documents × chunks per document.
Index memory = chunks × dimensions × 4 bytes, plus about half again for the graph and metadata.
One-off embedding work = chunks × tokens per chunk.
Serving load = peak questions per second, each needing one query embedding, one search, one rerank and one model call.

Why it matters

Estimates turn opinions into decisions: whether the index fits on one machine, whether you need compression, how long the first indexing run will take, and what the system will cost each month. Being roughly right, quickly, with the working shown, is what is being tested.

In the real world: Asked whether a single server can hold the index, a candidate works out the memory on the whiteboard in under a minute and answers 'yes, with a replica for safety'.

Visual example

Computed

Sizing one workload: 2 million documents, 8 chunks each, 1,024-number vectors, 50 questions a second at peak.

Estimation, step by step

quantity	working	result
Chunks	2 million × 8	16 million
Raw vector memory	16 million × 1,024 × 4 bytes	65.5 GB
With graph and metadata (×1.5)	65.5 GB × 1.5	98.3 GB
If stored as 1 byte per number	65.5 GB ÷ 4	16.4 GB
Tokens to embed once	16 million × 400	6.4 billion
Model calls per day at peak rate	50 × 86,400	4.3 million

Conclusion to state aloud: this fits one large machine plus a replica, or comfortably with 1-byte vectors.

Chasing precision. Round freely and state your assumptions aloud. An interviewer would rather see 'about 16 million chunks, about 65 GB' in twenty seconds than an exact figure in three minutes.

The trade-offs to name

What it is

Most RAG design choices are a trade between two things, and interviewers want both sides named. The recurring ones:

Chunk size: precise matching versus enough context.
Top-k: not missing the answer versus noise and cost.
ANN settings: recall versus speed.
Reranking: accuracy versus latency.
Query transformation: better retrieval versus extra model calls.
Freshness: up-to-date answers versus indexing cost.
Abstention: fewer wrong answers versus fewer answers.
Model size: quality versus cost and speed.

Why it matters

A choice stated without its cost sounds like a habit.

A choice stated with its cost, and the measurement that would settle it, sounds like engineering.
The pattern for any trade-off answer is: the two sides, what you would pick by default, and what you would measure to check.

In the real world: Asked 'would you add a reranker?', a strong candidate answers 'yes, if the latency budget has about 150 milliseconds to spare; I would confirm the gain by comparing MRR with and without it.'

Visual example

Trade-offs

Name both sides, pick a default, say what you would measure.

Eight trade-offs, each with its two sides

decision	more of this	costs you this	settle it by measuring
Chunk size	Larger: more context	Blurrier matching, bigger prompts	Recall@k at several sizes
Top-k into the prompt	Higher: less chance of missing	Noise, cost, latency	Answer quality versus k
ANN search effort	Higher recall	Slower queries	Recall against exact search
Reranker	Better order	100+ ms per question	MRR / nDCG with and without
Query transformation	Better retrieval	An extra model call	Recall with and without
Freshness	Faster updates	More indexing work	The stated requirement
Abstention threshold	Fewer wrong answers	More 'I don't know'	Both error rates
Model size	Better answers	Cost and latency	Answer metrics per rupee

Answering 'it depends' and stopping. Say what it depends on, then commit to a default: 'It depends on question type; for single-fact questions I would start at 400-token chunks and check recall@5 on the golden set.'

Common questions: fundamentals

What it is

The questions that open most RAG interviews. They test whether you can explain the idea plainly and place it beside its alternatives.

Why it matters

A crisp answer here earns the right to the harder questions. Each answer below is the two or three sentences an interviewer is listening for; all the detail is in the earlier modules.

In the real world: A phone screen for an AI engineer role opens with 'explain RAG to a product manager'; the candidates who progress are the ones who can do it without jargon.

Q: What is RAG and why use it?

Search your own documents for passages related to the question, put them in the prompt, and have the model answer from them. It gives the model private and current knowledge, reduces made-up answers, and lets every answer cite its source, without retraining the model.

Q: RAG or fine-tuning?

RAG for knowledge that is private, large or changing; fine-tuning for behaviour such as tone, format or a narrow skill. RAG can cite sources, be updated by editing a document and respect per-user permissions. They can be combined.

Q: Why not put everything in a long prompt?

For a small, stable collection, do. Beyond that: the collection does not fit, every question pays for every token, accuracy drops in very long prompts, and you cannot restrict what each user sees.

Q: What is an embedding, and how is similarity measured?

A list of numbers representing a text's meaning, produced so that similar texts get nearby vectors. Similarity is usually cosine similarity: the dot product divided by the two lengths. With normalised vectors it equals the dot product.

Q: Walk me through the two pipelines.

Indexing, offline: parse, clean, chunk, embed, store with metadata. Answering, online: rewrite the question, embed it, hybrid search, rerank, build the prompt, generate with citations.

Q: When is RAG the wrong choice?

Counting and totalling questions (use a database), questions needing the whole collection at once, tasks about style, and questions the model already answers well.

Answering 'what is RAG?' with a list of tools. The interviewer wants the idea in one sentence first; product names can come later, if at all.

Common questions: chunking, retrieval and ranking

What it is

The middle of the interview, where the interviewer checks that you have built one of these and know why each part is there.

Why it matters

These are the questions with the most room to show depth, so tie each answer to a failure it prevents.

In the real world: An on-site interviewer follows 'I'd use hybrid search' with 'give me a query each half would miss'; the candidate who answers 'a paraphrase, and an order ID' passes that round.

Q: How do you choose a chunk size?

It trades precise matching (small) against enough context (large). Start with structure-aware chunks of 300 to 500 tokens with a little overlap, then measure recall on a golden set at two or three sizes. Parent-child retrieval gets both: match small chunks, give the model the larger parent.

Q: Why hybrid search?

Keyword search (BM25) is exact on codes, names and rare terms and blind to paraphrase. Dense search matches meaning and blurs exact identifiers. Running both and fusing the ranks with reciprocal rank fusion covers both kinds of query.

Q: Bi-encoder versus cross-encoder?

A bi-encoder embeds the query and chunk separately, so chunks are precomputed and search is fast but less precise. A cross-encoder reads the pair together and is more accurate, but needs a model run per pair, so it only reranks the top 50 to 100.

Q: How does HNSW work?

It links each vector to its nearest neighbours in a layered graph. A search enters at the top layer, repeatedly moves to the neighbour closest to the query, and descends. More links (M) and a wider search (efSearch) raise recall at the cost of memory and speed.

Q: Pre-filtering or post-filtering?

Pre-filter. Post-filtering can leave too few or zero results when the filter is selective, and for permissions it lets restricted chunks become candidates at all.

Q: How do you handle follow-up questions?

Rewrite the latest message into a standalone question using the chat history, and search with that. Retrieve fresh on every turn.

Q: How would you improve retrieval without changing the model?

In order of value for effort: check the parsed text, fix chunking and add contextual headers, add keyword search for hybrid, add a reranker, add query rewriting. Measure each on the golden set.

Listing techniques without saying what each one fixes. 'I would add a reranker' is weaker than 'the right chunk is in the top 50 but not the top 5, so I would add a reranker.'

Practise it: Design a Semantic Search System, Design a Production RAG System

Common questions: evaluation, failures and production

What it is

The questions that distinguish someone who has run a RAG system from someone who has read about one.

Why it matters

Senior interviews spend most of their time here. The answers all come back to two habits: measure each half separately, and assume the documents and the users will both misbehave.

In the real world: A hiring manager asks every candidate to debug one wrong answer live; those who first ask 'was the right chunk in the context?' are the ones who get offers.

Q: How do you evaluate a RAG system?

Separately for each half, on a golden set of real questions. Retrieval: recall@k, MRR, nDCG. Generation: groundedness, answer relevance, correctness against a reference, using a judge model checked against human grades. Then online: feedback, escalations and A/B tests, with a regression gate on every change.

Q: How do you reduce hallucination?

Improve retrieval first, since weak context is the main cause. Then: a grounding instruction, a citation per claim with verification, abstention when nothing relevant is found, a low temperature, and a groundedness check on outputs.

Q: The system gives a wrong answer. How do you debug it?

Open the trace and ask whether the right chunk was in the prompt. If not, walk back: reranker, retriever, chunking, missing content. If it was, look at context order, distracting chunks, formatting and the prompt.

Q: How do you keep the index fresh?

Incremental indexing: detect changes by events or content hashes, re-embed only changed documents, replace their chunks, and propagate deletions and permission changes.

Q: How do you enforce access control?

Store each document's permissions on its chunks and pre-filter every search by the user's groups, taken from the session. Never post-filter, and never rely on a prompt instruction.

Q: What about prompt injection?

Retrieved text can contain planted instructions. Mark documents as data, limit what the assistant can do, check outputs, control what is indexed, and assume some attempts will succeed.

Q: How do you cut latency and cost?

Measure per stage. Stream the answer, use small models for rewriting, run searches in parallel, rerank fewer candidates, send fewer chunks, cache repeated questions, and tier models by difficulty.

Q: When would you use an agent or a knowledge graph?

An agent when questions need several dependent lookups or several sources. A graph when questions are about relationships across many documents. Both cost more, so start with a fixed hybrid pipeline and add them when measured failures require it.

Answering 'how do you evaluate?' with only an end-to-end accuracy number. The follow-up will be 'and when it drops, how do you know why?', which needs the retrieval and generation split.

Practise it: Design a Document Intelligence Platform, Design a Semantic Search System, Design a Production RAG System, Design an Enterprise Knowledge Assistant

Red flags interviewers listen for

What it is

Statements that signal a candidate has only built a demo.

Naming tools before understanding the problem.
No evaluation plan, or 'I would look at some answers'.
Dense search only, with no keyword component.
No mention of permissions in an enterprise design.
Fixing everything in the prompt.
No plan for unanswerable questions.
One pipeline, with no account of how documents are updated or deleted.
Certainty with no trade-offs.

Why it matters

Each has a positive counterpart, and the counterparts together are a description of a strong answer: requirements first, a golden set, hybrid retrieval with reranking, pre-filtered permissions, diagnosis by stage, abstention, incremental indexing, and trade-offs stated with a default and a measurement.

In the real world: Two candidates propose similar architectures; the one who adds 'and here is how I would know it is working, and what I would check first when it is not' gets the higher level.

Visual example

Signals

The right-hand column is a checklist for your own answer.

The red flag and the answer that replaces it

red flag	say this instead
'I'd use <vector database>' as the opening line	'First, what do the documents and questions look like?'
'I'd check a few answers'	A golden set with retrieval and answer metrics
Dense search only	Hybrid search, fused by rank, then a reranker
No mention of permissions	ACLs on chunks, pre-filtered at search time
'I'd tune the prompt'	'Was the right chunk in the context?' then fix that stage
Always answers	Abstain when nothing relevant is retrieved
Index built once	Incremental updates, deletions, re-indexing plan
Every advanced technique at once	Simple baseline first; add what a measured failure needs

Overcorrecting by proposing every advanced technique at once. A design with an agent, a graph, HyDE and three rerankers, for a help-centre assistant, is its own red flag. Start simple and add complexity only where a stated failure needs it.