Evaluating a RAG System

A RAG answer is produced by two parts in sequence, and each can fail on its own.

• Retrieval evaluation asks: did the search bring back the chunks that contain the answer?
• Generation evaluation asks: given those chunks, did the model write a correct, grounded answer?

An end-to-end score, judging only the final answer, tells you something is wrong but not where.

The fix depends entirely on which half failed.

• Poor retrieval is fixed with chunking, hybrid search or reranking.
• Poor generation is fixed with the prompt, the context layout or the model.
• Without separate scores, teams spend weeks rewriting prompts for a problem that lives in the search step.
• This is the single most important idea in RAG evaluation.

A golden set is a fixed list of test questions, each paired with the correct answer and with the IDs of the chunks or documents that contain it.

• Fifty to a few hundred questions is enough to start.
• They should come from real users where possible, and cover the range of real traffic: easy and hard, short and long, exact codes and loose paraphrases, and questions that have no answer in the documents.

It turns 'I think it got better' into a number.

• Every proposed change is run against the same questions, and the scores are compared.
• It is also the first thing an interviewer wants to hear when they ask how you would improve a RAG system: before changing anything, build the set that lets you measure the change.

Two retrieval metrics computed on the top k results.

• Recall@k: of all the relevant chunks that exist, what fraction appeared in the top k?
• Precision@k: of the k chunks returned, what fraction were relevant?

A simpler cousin, hit rate@k, asks only whether at least one relevant chunk made the top k.

Recall@k is the one that matters most for RAG.

• If the chunk holding the answer is not retrieved, the model cannot answer, so recall sets the ceiling for the whole system.
• Precision matters for the chunks that finally go into the prompt, where irrelevant ones cost tokens and distract the model.
• The first stage is tuned for recall, the reranker for precision.

Two metrics that care about position, not only presence.

• MRR (mean reciprocal rank): for each question take 1 / position of the first relevant result (1 if it is first, 0.5 if second, 0 if absent), then average over all questions.
• nDCG (normalised discounted cumulative gain): each relevant result earns credit that shrinks the lower it is placed, 1 / log2(position + 1); the total is divided by the best total possible, giving a score from 0 to 1.

Recall says the right chunk was found; these say how high it was placed.

• That matters because only the top few reach the prompt, and because models use earlier chunks more reliably.
• MRR suits questions with one right chunk. nDCG suits questions with several relevant chunks, and can also handle degrees of relevance (very relevant, partly relevant).
• They are the natural metrics for judging a reranker.

Groundedness, also called faithfulness, measures whether the answer is supported by the retrieved context.

• The usual method: break the answer into individual claims, check each claim against the context (with a judge model), and report the fraction that are supported.
• A score of 1.0 means nothing in the answer goes beyond the documents.

It is the direct measure of hallucination in a RAG system, and it needs no reference answer, so it can run on live traffic.

• Note what it does not measure: a faithful answer can still be wrong if the retrieved context was wrong or off-topic.
• Groundedness checks the answer against the context, not against the truth.

• Answer relevance asks whether the answer actually addresses the question that was asked, with no reference needed. An answer can be perfectly grounded and still fail here, by responding to a different question or burying the point.
• Answer correctness compares the answer with the reference answer in the golden set: does it state the same facts?

Together with groundedness these cover the generation side.

• Grounded but irrelevant: the model summarised the context without answering.
• Relevant but ungrounded: it answered from memory.
• Correctness is the measure closest to what users care about, but it needs reference answers, so it only works on the golden set.

Two measures of the retrieved context that can be estimated with a judge model, without hand-labelled chunk IDs.

• Context precision: what share of the retrieved chunks are actually relevant to the question, with more credit when the relevant ones are ranked first.
• Context recall: what share of the facts in the reference answer can be found in the retrieved chunks.

They complete a widely used set of four, popularised by the open-source RAGAS framework: context precision and context recall for the retriever, faithfulness and answer relevance for the generator. Naming these four and saying which half each one tests is a complete answer to 'how do you evaluate RAG?'.

LLM-as-judge means using a language model to grade outputs: 'does this context support this claim?', 'does this answer match the reference?'. The judge gets a clear rubric, the material to grade, and is asked for a verdict, ideally a simple one such as yes or no, with a short reason.

It is the only way to grade thousands of free-text answers at reasonable cost.

• But a judge is a model and makes mistakes, so it has to be checked before it is trusted: have people grade a hundred or so examples, compare with the judge's verdicts, and adjust the rubric until they agree most of the time.
• An unchecked judge gives numbers that look precise and mean little.

When there are no real user questions yet, a language model can write them.

• Give it a chunk and ask for a question that this chunk answers; the chunk is then the known relevant result.
• Repeating this over a sample of the collection produces a test set with labels for free.

It gets a project from zero to a measurable baseline in an afternoon, and it can cover parts of the collection that real users have not yet asked about. It is a starting point to be replaced or mixed with real questions as soon as traffic exists.

Three checks beyond the offline test set.

• User feedback: thumbs up or down, whether the user rephrased the question, whether they asked for a human.
• A/B tests: send a share of live traffic to a new version and compare outcomes.
• Regression gates: run the golden set automatically on every change and block the release if a key metric drops.

The golden set shows how the system does on questions you thought of; live traffic shows how it does on the questions people really ask. Each feeds the other: failures seen in production become new entries in the golden set, so the same mistake cannot return unnoticed.