LLM System Design · Module 14 of 17 · 9 concepts · about 13 min

Evaluation and Quality

How to measure the quality of a system whose answers differ every time: test sets, checks in code, model judges, how many examples are enough, and measuring on live traffic.

At work: What lets a team change a prompt or a model on a Tuesday afternoon and know by the evening whether it made things better or worse.

Why LLM systems need test sets

What it is

Ordinary software is tested by checking that an input gives one exact output. That does not work here:

the same input can produce different wording each time;
many different answers are equally correct;
a change that fixes one kind of question can quietly break another.

So quality is measured as a score across a set of examples, not as pass or fail on one.

Why it matters

Without a test set, every change is judged by trying a few questions and deciding it 'looks better'.

That is how teams ship a prompt that improved the three questions they tried and worsened thirty they did not.
A fixed set of examples, run before and after every change, turns opinion into a number that can be compared.

In the real world: A team that had argued for a week about which of two prompts was better settles it in an hour by running both on two hundred real questions.

Evaluating by demo. A handful of hand-picked questions always goes well, because they are the ones the builder had in mind. The value of a test set is in the cases nobody was thinking about.

Regression test set

What it is

A regression test set (or golden set) is a fixed collection of examples run on every change. Each example has:

an input: a real question, with any context it needs;
what a good answer must contain, or a reference answer;
a label for its kind (refund, delivery, out of scope, attack).

It should cover ordinary cases, hard cases, things the system should refuse, and every failure seen in production.

Why it matters

It is called a regression set because its main job is catching things that used to work and no longer do.

Every production failure that is fixed gets added, so the same mistake cannot return unnoticed.
Labels let the score be split by kind, which shows that a change helped refunds and hurt delivery questions, where a single overall number would hide it.

In the real world: A support platform's test set starts with eighty questions taken from real tickets and reaches six hundred within a year, each addition a failure somebody found.

[
  { "id": "refund-014", "kind": "refund",
    "input": "order arrived 40 min late, do i get anything back",
    "must_include": ["delivery fee", "refund"],
    "must_not_include": ["coupon"] },

  { "id": "scope-003", "kind": "out_of_scope",
    "input": "write me a poem about the monsoon",
    "expect": "polite_refusal" }
]

Building the set from invented questions. Real users are terser, messier and stranger than anything a team writes. Take examples from real logs wherever possible, and treat the set as something that keeps growing.

Practise it: Design an LLM Evaluation Platform

Deterministic assertions

What it is

Deterministic assertions are checks written in ordinary code, which give the same result every time:

the output is valid JSON and matches the schema;
it contains, or does not contain, a given phrase or value;
a number is within range, a date is valid, totals add up;
every cited source exists in the context that was provided;
the answer is under a length limit.

Why it matters

They are free to run, instant and never wrong about what they check.

They cannot judge whether an explanation is helpful, but a surprising share of real failures are things code can catch: a missing field, a wrong format, an invented citation, a forbidden promise.
Use code for everything code can check, and keep model judges for what is left.

In the real world: Half of a team's evaluation is plain assertions that run in seconds on every commit; the model-judged half runs nightly because it costs money and takes an hour.

def check(example: dict, answer: str) -> list[str]:
    """Returns the problems found. An empty list means these checks pass."""
    problems = []
    for phrase in example.get("must_include", []):
        if phrase.lower() not in answer.lower():
            problems.append(f"missing: {phrase}")
    for phrase in example.get("must_not_include", []):
        if phrase.lower() in answer.lower():
            problems.append(f"forbidden: {phrase}")
    if len(answer.split()) > 120:
        problems.append("too long")
    return problems

Reaching for a model judge first. It is slower, costs money, and can be wrong. If a rule can be written as code, write it as code.

Practise it: Design an LLM Evaluation Platform

LLM-as-judge

What it is

LLM-as-judge uses a model to grade outputs that code cannot: is this answer helpful, correct, supported by the sources? The judge is given:

the input and the answer to grade;
a rubric: exactly what counts as a pass, with examples;
an instruction to give a short reason and then a simple verdict.

Why it matters

It is the only practical way to grade thousands of free-text answers.

It works best when the question put to it is narrow and the verdict is simple. 'Is every claim in this answer supported by the context?
Yes or no' is a question a model answers reliably. 'Rate the quality from 1 to 10' is not.

In the real world: A team replaces its 1-to-5 'quality' judge with three separate yes-or-no judges (supported, complete, polite) and finds the results suddenly agree with what its own reviewers think.

You are grading a customer-support answer.

Question:   {question}
Policy:     {context}
Answer:     {answer}

Does the answer state only things the policy supports?
First give one sentence of reasoning.
Then answer exactly PASS or FAIL.

Asking for a score out of ten. The same answer gets a 6 one run and an 8 the next, and nobody can say what a 7 means. Use pass or fail, or a few clearly described levels.

Practise it: Design an LLM Evaluation Platform

Judge-human agreement

What it is

A judge is itself a model and makes mistakes, so it has to be checked before it is trusted:

people grade a sample, say a hundred examples;
the judge grades the same sample;
judge-human agreement is the share on which they match;
the rubric is revised until agreement is high, and re-checked now and then.

Why it matters

A judge that has never been compared with people gives numbers that look exact and may mean nothing.

Known weaknesses, which a rubric and a review must guard against, include preferring longer answers, preferring answers that sound like its own, and being swayed by the order in which options are shown.
This guarding is called judge bias mitigation.

In the real world: A judge that agreed with human graders only 68 percent of the time is brought to 93 percent by rewriting the rubric with four worked examples, and only then used to block releases.

Using the same model to produce and to judge without checking. A model tends to rate its own style highly. Check agreement with people, and where possible use a different model as the judge.

Practise it: Design an LLM Evaluation Platform

Pairwise comparison

What it is

Pairwise comparison shows a judge two answers to the same question, from two versions of the system, and asks which is better:

version A's answer and version B's, without saying which is which;
the judge picks one, or declares a tie;
the result is the share of questions each version wins.

Why it matters

Choosing between two things is an easier and more consistent judgement than scoring one thing on its own, for people and for models alike. It suits the most common question in practice, which is not 'how good is this?' but 'is the new version better than the old one?'.

In the real world: Deciding between two models, a team runs a pairwise comparison on three hundred questions; the new model wins 58 percent, loses 31 and ties 11, which is enough to switch.

Always putting the new version in the same position. Judges lean towards one position. Swap the order at random for each example, or judge each pair both ways round.

Practise it: Design an LLM Evaluation Platform

Confidence intervals and sample size

What it is

A score measured on a sample is an estimate, and it comes with a margin of error. For a pass rate p measured on n examples:

the margin, at 95 percent confidence, is about 1.96 × √(p × (1 − p) ÷ n);
the confidence interval is the score plus or minus that margin;
a larger set gives a narrower interval.

Why it matters

It tells you whether a difference is real.

At an 80 percent pass rate on 100 examples, the margin is about 8 points either way.
An apparent improvement from 80 to 83 percent is well inside that, and may be nothing but chance.
The smallest difference a test can reliably detect is called the minimum detectable effect, and it shrinks only as the set grows.

In the real world: A team stops arguing over one-point swings on its hundred-question set once someone works out the margin is eight points, and expands the set to a thousand.

Visual example

Computed

How sure you can be of a measured 80% pass rate, by the size of the test set. Small sets cannot tell small differences apart.

95% margin of error = 1.96 × √(p × (1 − p) ÷ n)

examples in the set	margin of error	the true rate is probably between
50	±11.1 points	68.9% to 91.1%
100	±7.8 points	72.2% to 87.8%
400	±3.9 points	76.1% to 83.9%
1000	±2.5 points	77.5% to 82.5%

To halve the margin, the set must be four times larger.

Announcing a two-point improvement measured on fifty examples. With a set that small the margin is over ten points. Either enlarge the set or say plainly that the result is inconclusive.

Practise it: Design an LLM Evaluation Platform

Evaluating on live traffic

What it is

Offline tests are followed by measurement on real use:

Shadow mode evaluation: the new version answers real requests in the background; its answers are graded, never shown.
A/B test: users are split at random between versions, and outcomes are compared.
Outcome metrics: for a support assistant, the containment rate (the share of conversations resolved without a person), plus how often users rephrase, give up or ask for an agent.
Sampling for review: a share of live answers is graded by the judges and by people every day.

Why it matters

A test set holds the questions you thought of.

Live traffic holds the ones you did not, and its mix changes over time.
Outcome metrics measure what the business actually wants, which a quality score only approximates.
Failures found live are added to the test set, so the two feed each other.

In the real world: A new prompt raises containment by four points and repeat contacts by six; the team concludes it is deflecting customers, not helping them, and rolls it back.

Visual example

Offline and online

Offline guards a release. Online tells you what to test next.

Where each kind of evaluation fits

check	when	tells you
Assertions in code	Every commit	Did anything basic break?
Judged regression set	Before every release	Is quality better or worse?
Shadow mode	Before exposing users	How does it behave on real requests?
A/B test	During rollout	Does it improve real outcomes?
Daily sampled review	Continuously	Is quality drifting?

Optimising a number that can be gamed. A support bot can raise its containment rate by making it hard to reach a human. Pair it with a measure of whether the problem was really solved, such as whether the customer contacts support again within a day.

Release gates

What it is

A release gate runs the evaluation automatically and blocks a change that makes things worse:

every change to a prompt, model, or retrieval setting triggers the test set;
results are compared with the current production version;
a drop beyond a set threshold, overall or in any labelled kind, stops the release.

Why it matters

It makes evaluation a habit instead of a favour.

People skip manual checks under deadline pressure; a gate does not.
Comparing by kind matters: a change can raise the overall score while breaking one category completely, and the gate should catch that.

In the real world: A model upgrade that improved the overall score is blocked because the 'should refuse' category fell from 96 to 71 percent; the gate caught what the average concealed.

Visual example

Release gate

Automatic, so that it is never skipped.

What happens to every change

Change proposedNew prompt, model or setting
Run the checksCode assertions, then judged tests
CompareWith production, per question type
Pass or blockA drop past the threshold stops it

Gating on the overall average alone. A two-point gain overall can hide a twenty-point collapse on refusals or on one language. Set thresholds per category for the categories that matter most.