Observability and Tracing

Ordinary software is debugged by reproducing the bug and stepping through the code. Agents break that method in five ways:

• Non-determinism: the same input can take a different path each run.
• The decision is not in your code: it is in a model's reply to a long prompt.
• Silent failure: a wrong answer raises no exception. It looks exactly like a right one.
• Distance: a mistake at step 3 shows up at step 12.
• Many moving parts: prompt, model version, tools, retrieved documents, memory and state all influence the outcome.

So you cannot rely on reproducing a failure. You must have recorded it.

• If the exact text sent to the model at the failing step was not saved, the cause is usually unrecoverable.
• Logs written for ordinary software ('request received', 'response sent') say nothing about why the agent chose what it chose.
• Observability for agents means capturing every step's inputs and outputs, so any run can be read afterwards like a story.

Two terms from distributed tracing:

• A trace is the complete record of one run, from request to final answer. It has a unique trace id.
• A span is one timed operation inside the trace: a model call, a tool call, a retrieval, a guardrail check. Each has a start, a duration, attributes and a parent.

Spans nest. The run is the root span; each agent step is a child; each model and tool call is a child of its step. The result is a tree you can expand and read.

The tree shape matches how an agent works.

• One view shows the whole run, where the time went and where it failed.
• Expanding a span shows exactly what went in and what came out.
• The trace id links everything across services, and across agents in a multi-agent system.
• OpenTelemetry is the open standard for this; using it means your traces work with most tools.

For every model call:

• the full prompt: system prompt version, messages, tool definitions;
• the full response, including tool requests;
• model and settings; input, output and cached tokens; cost; latency and time to first token; the stop reason.

For every tool call: name, inputs, result (or a reference if large), duration, success or error, and the idempotency key. For the run: user and tenant, agent version, step count, how it ended, total cost and time, and any human decisions.

The rule is: record enough that you could replay the decision.

• The prompt is the most important thing to save and the most often skipped, because it is long.
• Token and cost figures per step are what make cost debugging possible.
• The stop reason and the 'ended' reason distinguish a real answer from a limit being hit.
• Version stamps tie a change in behaviour to a change you made.

Metrics are numbers aggregated over many runs. The ones an agent team watches:

• Outcome: task success rate, escalation rate, reversal rate (actions a person had to undo).
• Efficiency: steps per task, tokens per task, cost per task, cost per resolved task.
• Latency: time to first response, time per step, total task time, at the median and at the slow tail (p50, p95).
• Failures: tool error rate per tool, invalid tool calls, runs ended by a limit, guardrail blocks, model errors and fallbacks.
• Health: queue wait time, rate-limit hits, cache hit rate.

Traces explain one run; metrics show the pattern.

• A rise in steps per task is the earliest sign that something has got worse, before success rate moves.
• 'Ended by step limit' creeping up means the agent is getting stuck.
• Cost per resolved task, not cost per task, is the business number: a cheaper agent that fails more is not cheaper.
• Slicing by agent version, tool, tenant and task type turns a vague 'it feels worse' into a located cause.

Four kinds of signal, each answering a different question:

• Traces: what happened in this run?
• Metrics: how are all runs doing, and is it changing?
• Logs: discrete events with detail: an error, a policy block, a deployment.
• Online evaluations: automated quality scores on a sample of live runs: was the answer supported, was the right tool used?

And one human signal: feedback such as thumbs up or down, corrections, and reviewers' verdicts, attached to the trace.

Ordinary monitoring tells you the service is up. It cannot tell you the answers are right.

• An agent can return HTTP 200 on every request while being wrong on a fifth of them.
• Online evaluations and user feedback are the only signals of quality in production.
• Linking them to traces means every bad score has a run you can open.
• Bad runs then become test cases, which is how the system improves (module 29).

Alerts turn metrics into action. Useful ones for agents:

• success rate or groundedness score falls below a threshold;
• cost per task, or total spend per hour, jumps;
• share of runs ended by a limit rises;
• one tool's error rate spikes;
• any run exceeds a hard ceiling on cost or steps;
• a guardrail's block rate changes sharply.

When an alert fires or a complaint arrives, the workflow is: find the affected traces, read one from the top, locate the first wrong step, classify the cause (prompt, tool, retrieval, memory, model), fix, and add the case to the test set.

The discipline is to find the first wrong step, not the last.

• The visible failure is usually downstream of the real mistake.
• Classifying the cause prevents the reflex of editing the prompt for what is actually a tool problem.
• Adding the case to the test set is what stops the same bug returning with the next change.

Recording everything has its own costs. Three controls:

• Redaction: remove or mask personal and secret data before a trace is stored: card numbers, addresses, tokens.
• Access and retention: restrict who can read traces, and delete them after a set period.
• Sampling: keep full detail for every failed, expensive, flagged or escalated run, and for a percentage of the rest.

Traces contain whatever your users typed and whatever your tools returned.

• That makes the trace store one of the most sensitive databases you run.
• Full prompts for every run at high volume are large; sampling keeps the useful ones.
• 'Keep all failures, sample successes' preserves almost all debugging value for a fraction of the storage.
• Summary metrics should still be recorded for every run, even those whose full trace is dropped.

The questions interviewers ask about observability and tracing, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.