Bezen

LLM System Design · Module 15 of 17 · 8 concepts · about 11 min

Observability and Operations

What to record about every request, what to watch on a dashboard, what to be woken up for, and how to handle model changes and incidents once a feature is live.

At work: The difference between learning about a problem from a dashboard at ten in the morning and learning about it from a customer's screenshot.

What to observe

What it is

An ordinary service is watched for three things: is it up, is it fast, is it erroring. An LLM system needs two more:

  • Is it up? Error and fallback rates.
  • Is it fast? Time to first token and total time.
  • What is it costing? Tokens and rupees, by feature.
  • Is it good? Quality scores on sampled answers.
  • Is it safe? Guardrail blocks, refusals, suspected attacks.

Why it matters

The dangerous failures of an LLM system produce no errors.

  • The service returns a success code, quickly, containing a wrong answer.
  • Uptime and latency dashboards stay green while quality falls.
  • So quality and cost have to be measured and displayed with the same seriousness as uptime, or nobody will see them move.

In the real world: A retailer's assistant runs for three weeks with perfect uptime while quietly giving out a retired returns policy; the first signal is a complaint, because nothing was watching answer quality.

Monitoring only what the infrastructure team already knows how to monitor. A system can be available, fast and wrong, and that combination is the one that reaches customers.

Tracing a request

What it is

A trace is the full record of one request as it passes through every stage. It is made of spans, one per stage, each with its timing and details:

  • the input check and its verdict;
  • what was retrieved, and from where;
  • the exact prompt that was built, and its version;
  • the model called, the attempt number, any fallback;
  • each tool call and its result;
  • the output check and the final answer.

One trace ID ties the spans together.

Why it matters

When a user reports a wrong answer from last Tuesday, the trace is the only way to see what the model was actually given.

  • The documents, the prompt version and the model may all have changed since.
  • With the trace, diagnosis takes minutes: was the right document retrieved? Was the prompt what you expected? Did it fall back to the weaker model?

In the real world: An engineer handed a complaint pulls the trace, sees the order lookup returned nothing and the model answered anyway, and fixes the template to refuse when the order is missing.

trace 7f3a...   total 4,310 ms   cost ₹1.31   user u_2291
  ├─ input_guardrail        62 ms   verdict=allow
  ├─ build_context         118 ms   3 passages, order 88213
  ├─ prompt                  4 ms   template=support v14
  ├─ model_call          4,060 ms   alias=support-chat
  │    ├─ attempt 1  provider-a/medium   timeout at 5,000 ms
  │    └─ attempt 2  provider-b/medium   3,700 in / 290 out
  └─ output_guardrail       66 ms   verdict=allow

Logging the question and the answer and nothing between. The fault is nearly always in the middle: the wrong passage, an empty variable, a fallback nobody noticed.

What to log for each model call

What it is

Every model call records a standard set of fields:

  • who: team, feature, user or tenant, environment;
  • what: model alias, the real model and version, prompt version, settings;
  • size: input and output tokens, cached tokens;
  • time: time to first token, total time;
  • outcome: success, error type, retries, whether it fell back;
  • cost: computed from the tokens and that model's prices;
  • content: the prompt and answer, stored according to their sensitivity.

Why it matters

These fields are what every later question is answered from. 'Why is the bill up?' needs tokens by feature. 'Did the new prompt help?' needs quality by prompt version. 'Was that slow answer a fallback?' needs the outcome. Fields that are not recorded at the time cannot be recovered afterwards.

In the real world: A cost spike is traced in ten minutes to one feature whose average input had doubled, because input tokens per feature were already on a chart.

Recording the alias but not the real model that answered. After a fallback or a canary, 'support-chat' may have been served by three different models in one hour, and without the real one logged the results cannot be told apart.

Metrics and dashboards

What it is

Logs are per request; metrics are the same data summarised over time. The core set:

  • latency: time to first token and total, at p50 and p95;
  • volume: requests and tokens per minute, against the rate limit;
  • errors: rate by type, retry rate, fallback rate;
  • cost: per feature, per day, per request;
  • caching: hit rate;
  • safety: guardrail block rate, refusal rate;
  • quality: sampled judge scores, user feedback, handover rate.

Why it matters

A dashboard answers 'is anything different from yesterday?' at a glance.

  • Each metric is split by feature, model and prompt version, because an overall average hides a problem confined to one of them.
  • Tokens per minute plotted against the rate limit is the capacity gauge: it shows trouble coming before the first rejected request.

In the real world: A platform team's main screen shows four rows (speed, errors, cost, quality) by feature; a jump in the fallback rate one morning reveals a provider problem before the provider announces it.

Visual example

Dashboard

Split every one by feature, model and prompt version.

The core metrics and what a change in each means
metrica rise usually means
Time to first token (p95)Provider slow, longer prompts, or a cold cache
Tokens per requestPrompt or history has grown; cost will follow
Fallback rateThe first-choice provider is in trouble
Cache hit rate (a fall)A changing value crept into the start of the prompt
Guardrail block rateAn attack, or a filter that has started over-blocking
Handover to a humanAnswers have got worse, or questions have changed

A dashboard of averages. Average latency can stay flat while the slowest five percent double. Plot percentiles for time, and rates for everything else.

Alerting

What it is

An alert pages a person. It is kept for things that need someone now:

  • error or fallback rate above a threshold for several minutes;
  • time to first token far above normal;
  • tokens per minute close to the rate limit;
  • spend running well ahead of budget;
  • a sudden jump in guardrail blocks;
  • a sustained fall in a sampled quality score.

Why it matters

Alerts are set on what users experience (answers are failing, answers are slow), not on every internal wobble.

  • A single retried request is normal; five percent of requests failing for five minutes is not.
  • Cost and quality alerts matter as much as outage alerts here, because a runaway bill or a silent quality drop can go on for days without one.

In the real world: A spend alert fires on a Saturday when a retry loop in a batch job starts burning through budget; it is stopped within the hour, at a cost of thousands, not lakhs.

Alerting on everything. A team that is paged twenty times a day for things that fix themselves learns to ignore pages, including the one that matters. Every alert should mean a person needs to act.

Quality drift

What it is

Drift is quality changing over time with no release on your side. It comes from three directions:

  • the model changes: the provider updates what a model name points to;
  • the questions change: a new product launches, a season starts, users ask new things;
  • the data changes: documents go out of date, a source stops updating.

Why it matters

Nothing in your deployment history explains a drift, which makes it hard to notice and harder to diagnose. The defences are to pin exact model versions, to run the test set on a schedule and not only on releases, and to grade a daily sample of live answers so that a slow decline shows up as a trend line.

In the real world: A scheduled weekly evaluation shows a five-point drop with no code change; the cause is a new product line that customers are asking about and no document covers.

Running evaluation only when you change something. The system can change without you. A weekly scheduled run of the same test set is what catches the provider-side change.

Migrating to a new model

What it is

Moving a feature to a new model, by choice or because the old one is being retired, follows a fixed sequence:

  • run the test set on the new model with the existing prompt;
  • adjust the prompt where it fails, and run again;
  • shadow it on live traffic;
  • move a small share of users, then more;
  • keep the old model available for rollback until the new one has settled.

Why it matters

A new model is not a drop-in replacement.

  • Formats, refusals, verbosity and tool-calling habits all shift.
  • Teams with a test set and aliases do a migration in days.
  • Teams without discover, after the retirement date, that their feature no longer works, with no way back.

In the real world: A platform team keeps a calendar of every model version in use and its retirement date, and starts each migration two months early with the test set.

Visual example

Migration

The same gates as any release, plus a way back.

Moving a feature to a new model

  1. Test offlineRegression set on the new model
  2. Fix the promptAdjust for its habits
  3. Shadow0% of usersGraded on live requests, unseen
  4. Canary5% → 25% → all
  5. Keep the old oneUntil the new one runs cleanly

Leaving it to the retirement date. Providers announce retirements months ahead. Track those dates, and treat a migration as planned work, not an emergency.

Handling an incident

What it is

When a live feature misbehaves, the first actions are quick switches that need no code release:

  • roll back the prompt or the model alias to the last good version;
  • fail over to another provider;
  • tighten a guardrail or turn off a tool;
  • switch the feature off, falling back to the non-AI experience.

Diagnosis from traces comes after the bleeding has stopped.

Why it matters

Because prompts, aliases and guardrail settings live in configuration, each can be changed in seconds.

  • That is one of the main practical benefits of the gateway and the prompt registry.
  • A team that must write, review and deploy code to stop a misbehaving assistant will be watching it misbehave for an hour.

In the real world: An assistant starts quoting a wrong price after a document update; the on-call engineer disables the pricing tool with one flag, and the assistant says it cannot quote prices until the document is fixed.

Having no off switch. Every AI feature needs a way to be disabled at once, with a sensible non-AI fallback behind it. The day it is needed is not the day to build it.