Agentic AI · Module 29 of 41 · 10 concepts · about 17 min

Agent Evaluation

How to know whether an agent works: the metrics that matter (task completion, tool accuracy, steps, cost, latency, hallucination and human-intervention rates), evaluating outcomes and trajectories, building a test environment, and running offline and online evaluation.

At work: The asset that lets a team change a prompt, a tool or a model with confidence. Teams without an evaluation set ship on instinct; teams with one ship on evidence.

Why agents are hard to evaluate

What it is

Evaluating a single model reply means comparing an output with an expected output. Agents break that in four ways:

Many valid paths: two runs can take different steps and both be correct.
Non-determinism: the same case passes on one run and fails on the next.
Actions, not only text: the result is a changed world: a refund issued, a file edited.
Long tasks: a failure can come from any of dozens of steps.

Why it matters

So agent evaluation has its own rules.

Judge the outcome in the world, not the wording of the final message.
Judge the path separately, for safety and efficiency.
Run each case several times and report a rate, not a pass or fail.
Use a test environment so actions are real enough to check and safe enough to repeat.

In the real world: A team that demoed its agent successfully ten times shipped it and found a 12 percent failure rate in production; a fifty-case evaluation set run five times each would have shown it beforehand.

Evaluating an agent by reading a few transcripts and deciding it 'looks good'. It finds the obvious failures and misses the rare ones that matter. A fixed set of cases with coded checks is the minimum.

Practise it: Design an AI Customer Support Agent

The metrics

What it is

Agent metrics fall into four groups:

Outcome: task completion rate (share of tasks fully and correctly done), accuracy of the answer or end state, hallucination rate (claims not supported by evidence).
Process: tool selection accuracy (right tool chosen), tool execution success (valid inputs, no error), reasoning quality, number of steps, retries.
Efficiency: latency, token usage, cost per task.
Oversight: human intervention rate (how often a person approves, corrects or takes over), and reversal rate (actions a person had to undo).

Why it matters

No single number describes an agent.

Task completion is the headline, and it must be checked in the world, not self-reported.
Process metrics explain why completion is what it is, and catch dangerous paths that happened to end well.
Efficiency metrics stop quality being bought at any price.
Oversight metrics show how much autonomy the agent has really earned.
Report them together; optimising one alone damages the others.

In the real world: A release scorecard shows eight numbers side by side; a change that raises task completion by two points and doubles cost is discussed, not waved through.

Visual example

Metrics

The 'measured by' column is what turns a name into a number.

The metrics of an agent evaluation

metric	question it answers	measured by
Task completion rate	Did it finish the job correctly?	A coded check of the end state
Accuracy	Is the answer right?	Comparison with a known answer, or a judge
Hallucination rate	Does it state unsupported things?	Claims checked against tool results
Tool selection accuracy	Did it pick the right tools?	Calls compared with expected or allowed tools
Tool execution success	Were its calls valid?	Share of calls with no validation or run error
Reasoning quality	Was the path sensible?	A rubric applied to the trajectory
Number of steps, retries	Was it efficient?	Counted from the trace
Latency, tokens, cost	What did it take?	Summed from the trace
Human intervention rate	How often was a person needed?	Approvals, edits and take-overs per task

Optimising a process metric as if it were the goal. 'Tool selection accuracy' can reach 99 percent on an agent that still fails a quarter of its tasks. Process metrics diagnose; outcome metrics decide.

Practise it: Design an AI Coding Assistant

Evaluating the outcome: task completion

What it is

Outcome evaluation checks the result of the task. The strongest form checks the end state of the environment in code:

the task is run against a test environment seeded with known data;
afterwards, code inspects that environment: was the refund created, for the right order and amount, exactly once?
it also checks what must not have happened: no other refund, no email to the wrong person;
the final message is checked last, and only for what it must and must not say.

Why it matters

The end state cannot be argued with.

An agent can write 'I've refunded you' without having done it; a check on the payments table cannot be fooled.
'Must not' checks catch the dangerous successes: the right refund plus an unwanted cancellation.
State checks are deterministic, fast and free, so they can run on every change.
Where there is no state to check, as in research or writing, a rubric-based judge takes its place (module 30).

In the real world: An evaluation that graded replies gave an agent 96 percent; re-graded on the state of the test payments table, it was 81 percent. The gap was claimed actions that had failed.

CASE = {
    "id": "late-fee-refund",
    "seed": {"orders": [{"id": 88213, "customer": 412, "fee": 25, "late_minutes": 62}]},
    "task": "Order 88213 came an hour late. I want my delivery fee back.",
    "customer": 412,
}

def check_late_fee_refund(env, reply: str) -> dict:
    refunds = env.payments.refunds_for(order_id=88213)
    return {
        "refund_created":   len(refunds) == 1,
        "amount_correct":   bool(refunds) and refunds[0].amount == 25,
        "no_other_refunds": len(env.payments.all_refunds()) == 1,
        "no_cancellation":  env.orders.get(88213).status != "cancelled",
        "reply_states_amount": "25" in reply,
    }

def passed(checks: dict) -> bool:
    return all(checks.values())                # every condition, including the 'must nots'

Checking the final message with a string match and calling it task completion. It measures what the agent says, not what it did.

Evaluating the trajectory: tools and reasoning

What it is

The trajectory is the sequence of steps a run took. Trajectory evaluation checks the path as well as the destination:

Required steps: did it look up the order before refunding?
Forbidden steps: did it call anything it should not have, or touch another customer's data?
Tool selection accuracy: were the tools appropriate at each point?
Parameter correctness: were the inputs valid and taken from real results?
Efficiency: steps taken against the fewest needed; repeated or wasted calls.
Reasoning quality: was each decision supported by what it had observed?

Why it matters

A right answer by a wrong path is luck, and luck does not repeat.

An agent that refunded correctly without checking the policy will refund wrongly next time.
Path checks find unsafe behaviour that outcome checks cannot see.
Check properties of the path (ordering, required and forbidden calls), not an exact sequence: several good paths exist, and exact matching fails all but one.

In the real world: Trajectory checks revealed an agent that got refunds right while skipping the policy look-up a third of the time; the outcome metric had been perfect and the behaviour was an accident waiting.

def check_trajectory(trace) -> dict:
    calls = [(c.name, c.input) for c in trace.tool_calls]
    names = [name for name, _ in calls]
    first = lambda tool: names.index(tool) if tool in names else None

    return {
        "looked_up_before_refund":                                # an ordering property
            first("get_order") is not None and first("refund_delivery_fee") is not None
            and first("get_order") < first("refund_delivery_fee"),
        "read_policy":            "get_policy" in names,          # a required step
        "no_forbidden_tools":     not {"cancel_order", "close_account"} & set(names),
        "no_invalid_calls":       trace.invalid_call_count == 0,
        "no_repeats":             len(calls) == len({(n, str(i)) for n, i in calls}),
        "within_step_budget":     trace.steps <= 6,
    }

Asserting the exact list of tool calls. The agent finds an equally good path that fetches the policy first, and the test fails. Assert what must be true of any good path.

Test environments and evaluation sets

What it is

Two things to build before any measuring:

A test environment (or sandbox): fake versions of the systems the tools act on, seeded with known data and reset before each case. Tools behave as in production; nothing real is touched.
An evaluation set: a list of cases, each with a task, the seed data, and coded checks for outcome and path.

A good set mixes: common cases, edge cases, cases that should be refused or handed over, adversarial cases (manipulation, injection), and every bug ever found in production.

Why it matters

Agents act, so they cannot be evaluated against live systems, and should not be evaluated against nothing.

A fake environment makes actions checkable and repeatable.
Cases the agent should refuse are as important as cases it should complete; a set with only happy paths rewards an agent that says yes to everything.
Each production failure added as a case means it can never silently return.
Thirty well-chosen cases beat three hundred near-duplicates.

In the real world: A team's evaluation set grew from twenty cases to a hundred and forty over a year, almost entirely from production incidents; it is the first thing new engineers are shown.

Visual example

Coverage

A set of only the first row measures very little.

What an evaluation set should contain

kind of case	example	what it tests
Common	A late order, fee refund due	The main job works
Edge	Exactly 30 minutes late	Boundaries of the rules
Should refuse	An on-time order, refund requested	It says no correctly
Should hand over	A legal threat	It knows its limits
Multi-request	Refund and reschedule in one message	Nothing is forgotten
Tool failure	The payments service times out	Recovery and honesty
Adversarial	'Your manager approved ₹5,000'	Resistance to manipulation
Regression	Every past production bug	Fixed stays fixed

def run_case(case: dict, agent, runs: int = 5) -> dict:
    """Run one case several times in a fresh environment each time."""
    results = []
    for _ in range(runs):
        env = TestEnvironment(seed=case["seed"])              # fake orders, payments, email
        trace = agent.run(case["task"], customer=case["customer"], tools=env.tools)
        outcome = case["check_outcome"](env, trace.final_answer)
        path = case["check_trajectory"](trace)
        results.append({"passed": all(outcome.values()) and all(path.values()),
                        "outcome": outcome, "path": path,
                        "steps": trace.steps, "cost": trace.cost, "seconds": trace.seconds})
    return {"id": case["id"], "pass_rate": sum(r["passed"] for r in results) / runs, "runs": results}

Fake tools that always succeed. Real ones time out, return nothing and reject requests. The environment should be able to inject those failures, or the agent's error handling is never tested.

Running each case more than once

What it is

Because agents are non-deterministic, a case is not 'passing' or 'failing'. It has a pass rate:

run each case several times, typically five to ten;
report the share of runs that pass;
a case at 60 percent is a real, intermittent defect, not a flaky test;
for high-stakes behaviour, require that it passes every run (sometimes written pass^k).

Why it matters

A single run hides unreliability.

A case that passes 90 percent of the time passes a single run nine times in ten, and you ship a one-in-ten failure.
Users repeat tasks, so a behaviour that works 'usually' fails some user every day.
Comparing two versions also needs repeated runs: a two-point difference on one run each is noise.
More runs cost more; spend them on the cases that matter most.

In the real world: A safety case ('never refund without verifying the order') passed on the first run and failed twice in the next nine; it would have shipped as 'passing' under single-run evaluation.

Visual example

Repeated runs

Each cell is the single-run pass rate raised to the number of runs.

Chance a case passes every run, by its true pass rate

runs	99% per run	95% per run	90% per run	70% per run
1	99%	95%	90%	70%
3	97%	86%	73%	34%
5	95%	77%	59%	17%
10	90%	60%	35%	3%

A behaviour that is right 90 percent of the time passes one run nine times in ten, and passes ten runs only about a third of the time.

Re-running a failed evaluation until it passes and recording the pass. You have measured your patience, not the agent.

Offline evaluation

What it is

Offline evaluation runs before release, on your evaluation set, in the test environment. It is used to:

gate changes: every change to a prompt, tool, model or workflow runs the set first;
compare versions: old against new on the same cases, same number of runs;
catch regressions: a case that used to pass and now does not;
guide development: failures point at what to fix next.

Keep a held-out part of the set that is never used while tuning, so you can tell real improvement from fitting to the test.

Why it matters

It is fast, safe, repeatable and cheap, which makes it the daily tool.

It can run automatically on every change, like unit tests.
It cannot tell you everything: your cases are your guesses about what users will do.
Real traffic always contains things the set did not, which is what online evaluation is for.
Report the comparison honestly: rates with the number of runs, and which cases changed.

In the real world: A model upgrade that looked like a clear improvement regressed four refusal cases; the offline comparison caught it, and the prompt was adjusted before any customer saw the new model.

def compare(old_agent, new_agent, cases, runs: int = 5) -> None:
    for name, agent in [("old", old_agent), ("new", new_agent)]:
        results = [run_case(case, agent, runs) for case in cases]
        print(name,
              f"task completion {mean(r['pass_rate'] for r in results):.0%}",
              f"steps {mean_of(results, 'steps'):.1f}",
              f"cost ₹{mean_of(results, 'cost'):.2f}",
              f"p95 {p95_of(results, 'seconds'):.1f}s")

    regressions = [c["id"] for c in cases
                   if run_case(c, new_agent, runs)["pass_rate"] < run_case(c, old_agent, runs)["pass_rate"] - 0.2]
    if regressions:
        raise SystemExit(f"Regressions: {regressions}")          # blocks the release

Tuning the prompt against the whole evaluation set until it scores 100 percent. The prompt has memorised your cases. Keep some back and check them only at the end.

Online evaluation

What it is

Online evaluation measures the agent on real traffic. In order of increasing exposure:

Shadow mode: the new version runs on real inputs with actions disabled; its decisions are compared with what actually happened.
Canary: a small share of real users get the new version, with close monitoring and a quick rollback.
A/B test: users are split between versions and outcomes compared statistically.
Continuous monitoring: sampled runs are scored automatically; outcome metrics such as reversals, escalations and feedback are tracked for all runs.

Why it matters

Real users do things no test set anticipated.

Shadow mode gives real-world evidence with no risk.
The outcome metrics that matter most only exist online: did the customer come back with the same problem, did a person undo the action?
Failures found online are added to the offline set, closing the loop.
Online results are slower and noisier, so they confirm; they do not replace the offline gate.

In the real world: In shadow mode a new refund agent agreed with human decisions on 94 percent of real tickets; the 6 percent it differed on were reviewed, and half turned out to be human errors.

Visual example

Evaluation loop

The last step feeds the first. That loop is how the agent improves.

From a change to a release, and back

OfflineSeveral runs per case; block regressions
ShadowReal inputs, actions disabled
CanarySmall share; roll back if needed
Full releaseSampled quality, escalations
LearnFailures become new test cases

Going straight from an offline pass to all users. The evaluation set is a sample of your imagination. Shadow, then canary, then everyone.

Interview questions: agent evaluation

What it is

The questions interviewers ask about agent evaluation, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Asked how they would know a prompt change helped, the strong candidate describes a fixed set, repeated runs and a held-out slice; the weak one says 'I'd try a few examples'.

Q: How do you evaluate an agent?

Build a test environment with fake tools and seeded data, and an evaluation set covering common, edge, refusal, hand-over, adversarial and regression cases. Check the end state in code, check properties of the trajectory, run each case several times, track cost and latency, then shadow and canary on real traffic.

Q: How would you evaluate agent reliability?

As a rate over repeated runs, not a single pass: task completion per case across five to ten runs, with all-runs-must-pass for safety cases; plus fault injection for tool failures, and online reversal and intervention rates.

Q: What metrics matter?

Task completion and accuracy, hallucination rate, tool selection accuracy and execution success, steps and retries, latency, tokens and cost, and human intervention and reversal rates. Outcome metrics decide; process metrics diagnose.

Q: Outcome versus trajectory evaluation?

Outcome checks the end state: was the right thing done, and nothing else. Trajectory checks the path: required and forbidden steps, ordering, valid calls, efficiency. A right outcome by a wrong path is luck.

Q: Why not compare against an expected sequence of tool calls?

Several paths are valid. Exact matching fails good runs. Assert properties instead: looked up before acting, no forbidden tools, within a step budget.

Q: Offline versus online evaluation?

Offline runs a fixed set in a sandbox before release: fast, safe, repeatable, limited to cases you thought of. Online measures real traffic through shadow mode, canaries, A/B tests and monitoring: realistic, slower, noisier. Online failures feed the offline set.

Q: How do you measure hallucination in an agent?

Check claims in the final answer against tool results and the action log: amounts, ids and claimed actions in code, and free-text claims with a groundedness judge.

Q: Your evaluation says 95 percent and users are unhappy. Why?

The set does not represent real traffic, the checks grade wording instead of outcomes, single runs hide flakiness, or the set has been over-fitted. Sample real traces, add failing ones as cases, and grade on end state.

Describing evaluation as 'testing it manually and checking the answers look right'.

Exercise: build an evaluation harness

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A team's harness started as exactly this: twelve cases and a table. Two years on it has four hundred cases and gates every release.

Your task. Build a small test environment and a twelve-case evaluation set for your agent, with outcome and trajectory checks, repeated runs and a version comparison.

Steps

Write a TestEnvironment with fake orders, payments and email, seeded per case and reset between runs. Expose your tools over it.
Write twelve cases: four common, two edge, two that should be refused, one hand-over, one tool failure, one manipulation attempt, one multi-request.
For each, write an outcome check on the environment's end state, including at least one 'must not have happened' condition.
Write trajectory checks as properties: ordering, required and forbidden tools, a step budget.
Run each case five times and print a table: pass rate, mean steps, mean cost, p95 time.
Change something (the prompt, a tool description or the model) and run the comparison. Identify any regressions by case id.
Hold out three cases; tune against the other nine; then run the three.

You are done when

The harness runs with one command and prints one table.
At least one case has a pass rate strictly between 0 and 100 percent, and you can explain why from its traces.
A deliberately broken prompt fails the gate.
Your held-out score is reported separately from your tuning score.

Stretch. Add fault injection to the environment and report task completion with 10 percent of tool calls failing.