Interview: Scenario Questions

1. Clarify

• Is it every task or only some kinds? One tool or several?
• Did it start after a change: a deploy, a tool update, a model version, an API change upstream?
• What does the tool return on those calls?

2. Likely causes, most probable first

• The tool returns an empty result, a bare 'Error', or a huge payload in which the answer is buried, so the model tries again.
• The tool's error is not actionable: it does not say what was wrong or what to do instead.
• The upstream API changed or is failing, and the tool is hiding that.
• The task is impossible with the tools available, and the agent has no 'I cannot do this' route.
• The result was trimmed or compacted away, so the model no longer sees that it already called it.
• Rarely: the prompt insists on a step ('always verify the order') with no exit condition.

3. Confirm with evidence

• Open three failing traces. Read the tool's result on the first repeated call.
• Compare with a successful run of the same tool.
• Check the tool's error rate and response size over time for a step change.
• Call the tool directly with the same arguments.

4. Fix

• Contain: the step and cost limits are already bounding the damage; add repeat detection that stops after two identical calls and hands over.
• Repair the tool's output: explicit 'not found' text, an error that says what to try instead, and a size cap.
• If upstream is failing, surface that plainly and add a circuit breaker.
• Give the agent a legitimate exit: an ask_user or hand_over tool, described as a correct outcome.

5. Prevent it recurring

• A regression case that injects that failure and asserts the run ends by hand-over within three steps.
• A metric and alert on runs ended by the step limit, and on repeated identical calls.
• A rule for every tool: no empty results, no bare errors.

1. Clarify

• How many, how much, since when? Is it still happening?
• Which kind of error dominates: over the amount, ineligible orders, or duplicates? They have different causes.
• What changed: policy, prompt, model, a tool, traffic, or a new manipulation doing the rounds?

2. Likely causes, most probable first

• Eligibility and limits are stated in the prompt and not enforced in the refund tool.
• Duplicates: retries after timeouts, or resumed runs, with no idempotency key.
• The amount comes from the model's reasoning rather than from the order record.
• Customers have found a phrasing that persuades the model ('a manager approved it').
• A policy document or tool result changed, and the agent is applying an outdated or misread rule.
• The order status was read early and was stale by the time of the refund.

3. Confirm with evidence

• Pull the traces of twenty wrong refunds and classify each by cause.
• For duplicates, check whether the two calls shared a run and whether the tool received a key.
• For wrong amounts, compare the amount requested with the order total in the tool's input.
• Check which version of prompt, tools and model each bad run used.

4. Fix

• Contain immediately: disable the refund tool, or drop every refund to 'requires approval'. The agent keeps answering questions and hands refund requests to people.
• Reconcile: list affected customers from the audit log and decide on recovery.
• Redesign the tool: it takes an order id and a reason, computes the eligible amount itself from the order and policy, enforces caps and a daily count, and requires an idempotency key.
• Fix the flow: a workflow in which the payment step is reachable only after a coded verification step; approval above a threshold, triggered by code.
• Verify the reply against the ledger before it is sent.

5. Prevent it recurring

• Every bad refund becomes an evaluation case asserting on the payment ledger's end state, run several times.
• Adversarial cases for false authority and pressure, required to pass every run.
• Alerts on refund amount per hour, refunds per customer and the reversal rate.
• Release changes through shadow mode and a canary, with a kill switch per tool.

1. Clarify

• Is it all tasks or a few very expensive ones?
• What was deployed or changed yesterday, on our side or upstream?
• Did success rate, steps per task or latency change at the same time?

2. Likely causes, most probable first

• A tool started returning much larger results, which are resent on every later step.
• Prompt caching stopped working: something changed in the prefix, such as a timestamp, a reordered tool list or a new tool.
• Steps per task rose: a failing tool causing retries, or loops.
• A routing change sends easy requests to the large model.
• A handful of runaway runs with no effective budget.
• Compaction stopped triggering, so contexts grow without limit.
• Abuse: one account generating very long tasks.

3. Confirm with evidence

• Compare yesterday and today on: tokens per task split into input, output and cached; steps per task; cost by model; cost by tool; the ten most expensive runs.
• Check cache-read tokens: if they fell to zero, diff the prefix between two requests.
• Check each tool's average result size for a jump.
• Look at the cost distribution: a shifted median means a systematic cause; a fat tail means runaways.

4. Fix

• Contain: a per-run cost budget and a per-user limit, if they are missing or too high.
• Then the specific repair: cap or trim the tool result; restore the stable prefix; fix the failing tool; correct the routing.
• Re-run the evaluation set to confirm the fix did not change quality.

5. Prevent it recurring

• Alerts on cost per task, cache hit rate, steps per task and hourly spend.
• A size cap on every tool result.
• A test that the request prefix is byte-identical across two calls.
• Cost per task reported in the evaluation table for every release.

1. Clarify

• How is the 60 percent measured, and on what?
• Which kinds of task fail? Is it uniform or concentrated?
• How do production tasks differ from the demo: length, phrasing, number of requests per message, data quality?

2. Likely causes, most probable first

• Real tasks are longer, so per-step errors compound.
• Real inputs are messier: several requests in one message, missing ids, typos, other languages.
• Real tools fail: timeouts, empty results, rate limits the demo never met.
• Real data has cases the demo data lacked: partial orders, cancelled items.
• There was never an evaluation set, so nobody knew the true rate.
• Context grows on long tasks and the agent loses the goal.

3. Confirm with evidence

• Sample a hundred failed production traces and classify the first wrong step of each.
• Rank the categories; usually three or four account for most failures.
• Compare steps per task for successes and failures.

4. Fix

• Build the evaluation set from those real failures, with end-state checks, and establish the baseline.
• Fix the top category first, then re-measure. Typical fixes: tool results and errors; a plan and completeness check for multi-part requests; context trimming; a clarifying question when an id is missing.
• Narrow the scope meanwhile: route the task kinds it fails to people, and let it handle what it does well.

5. Prevent it recurring

• Every production failure category has cases in the set; every release runs it.
• Shadow mode and a canary before any expansion of scope.
• Dashboards for completion, escalation and steps per task by task kind.

1. Clarify

• What data, to where, by what channel: a tool call, a fetched URL, a rendered image or link?
• What tools and data did that agent have at the time?
• Is it reproducible, and has it been exploited?

2. Likely causes, most probable first

• The page contained hidden instructions, which entered the context through the fetch tool.
• The same agent held private data and an outbound channel: the three-part combination that makes exfiltration possible.
• The outbound channel may be subtle: a URL fetch with data in the address, or a markdown image the client loads automatically.
• No confirmation was required for outbound actions.

3. Confirm with evidence

• Reproduce in a test environment with a planted page and a fake secret.
• Read the trace: the tool result containing the instruction, then the action that carried the data out.
• Search traces for the same destination to gauge exposure.

4. Fix

• Contain: disable the outbound capability, or web reading, for agents that hold private data; block the destination; stop rendering model-written images and links to unapproved hosts.
• Redesign: remove one leg. A tool-less reader summarises untrusted pages into a typed object; the agent with private data and tools never sees raw page text.
• Allow-list outbound destinations; require user confirmation, showing the content, for sends.
• Notify affected users and follow the incident process.

5. Prevent it recurring

• Injection cases with planted pages in the evaluation set, asserting the harmful end state never occurs, on every run.
• A design rule reviewed for every agent: private data, untrusted content and an outbound channel may not meet without a control between them.
• Regular red-team exercises; alerts on outbound requests to new domains.

1. Clarify

• Did the tool run and fail, or was it never called?
• Is it one tool or several? Since when?
• What did the tool return in those runs?

2. Likely causes, most probable first

• The tool failed or timed out, and the model wrote the reply as if it had succeeded.
• The tool returned an ambiguous result that reads as success, such as an empty string or 'OK' with an error inside.
• The model never called the tool: it answered from intent ('I'll reschedule that') or hit the step limit before acting.
• An approval was pending or declined, and the reply did not reflect it.
• The action succeeded and a later step undid or overwrote it.

3. Confirm with evidence

• For reported cases, compare the reply with the action ledger and the system of record.
• In the trace, find the tool call and read its exact result.
• Count, across all runs, replies that claim an action with no matching successful call.

4. Fix

• An output check in code: any claimed action, amount or id in the reply must match a successful entry in the ledger; otherwise regenerate from the ledger or hand over.
• Make tool results unambiguous: 'Rescheduled to 6 pm tomorrow' or 'Failed: the slot is no longer available'.
• Build the closing summary from recorded outcomes, not from the model's memory of its plan.
• Prompt: state only what has been done; never promise.

5. Prevent it recurring

• Fault-injection cases where the tool fails and the reply must say so.
• A metric for claimed-but-unperformed actions, with an alert above zero.
• End-state checks, not reply checks, in the evaluation set.

1. Clarify

• Forty seconds to the first response, or to completion?
• Is it all tasks or the long ones? What is the p95?
• How many steps does a typical task take?

2. Likely causes, most probable first

• Many sequential model calls: independent look-ups done one per turn.
• A large model with high reasoning effort used for every step, including trivial ones.
• Long outputs at intermediate steps.
• Slow tools, or tools called repeatedly for the same data.
• Long contexts raising time to first token; no caching.
• No streaming, so the user sees nothing for the whole duration.
• Queue wait or rate-limit backoff under load.

3. Confirm with evidence

• Take the timing breakdown from traces: per step, model time against tool time against waiting.
• Count sequential steps that could have been parallel.
• Find the slowest tool and the largest outputs.

4. Fix

• Perceived latency first, because it is quick: stream status messages and the answer; first output in under two seconds.
• Parallel tool calls for independent look-ups; pre-load what most tasks need.
• A small fast model and low effort for routing and routine steps.
• Cache stable look-ups; trim context; enable prompt caching.
• A fast path for simple requests that skips the loop.
• For tasks that truly need a minute: acknowledge, run in the background, notify.

5. Prevent it recurring

• A latency budget per step, and p95 time to first response and to completion on the dashboard with alerts.
• Latency reported beside success and cost in the evaluation table for every release.

1. Clarify

• Was the evaluation set run before the switch? What did it show?
• Which behaviours changed: tool choice, refusals, verbosity, step count?
• Was anything else released at the same time?

2. Likely causes, most probable first

• The prompt contained emphasis or workarounds written for the old model, which the new one over-applies or interprets differently.
• The new model is more, or less, willing to act without asking, changing escalation behaviour.
• Tool-calling habits differ: more parallel calls, different argument formats.
• Default settings changed, such as reasoning effort or output length.
• The evaluation set did not cover the behaviours that regressed.

3. Confirm with evidence

• Run the full evaluation set on both versions, several runs each, and list regressions by case.
• Compare trajectories on regressed cases: where do the two models first diverge?
• Check the provider's migration notes for changed defaults.

4. Fix

• Contain: roll back to the previous model; that is why versions are pinned.
• Re-tune for the new model: remove old workarounds and heavy emphasis, restate rules plainly with reasons, set effort explicitly.
• Confirm that safety behaviours are enforced in code, so a model change cannot loosen them.
• Re-release through shadow mode and a canary.

5. Prevent it recurring

• Pin model versions; never take an upgrade implicitly.
• The evaluation set gates model changes exactly as it gates prompt changes.
• Add the regressed cases, especially refusals, as must-pass-every-run cases.

1. Clarify

• Are the contradictions about facts (different sources) or definitions (different date ranges, units)?
• What does a brief look like? What does a worker return?
• How does this compare with a single agent on the same questions?

2. Likely causes, most probable first

• Briefs are one line, with no scope boundary, so workers overlap and leave gaps.
• No shared definitions were fixed before the work fanned out.
• Workers return prose, so conflicts are invisible to code and smoothed over by the synthesiser.
• The synthesiser is told to 'combine', with no instruction to surface conflicts or gaps.
• Workers share a transcript, so contexts are bloated and each is influenced by the others' noise.
• The task does not actually split into five independent strands.

3. Confirm with evidence

• Read the briefs and reports for three bad runs side by side.
• Trace each contradictory figure to its worker and source.
• Measure tokens spent on inter-agent messages against tokens spent on work.
• Run a single agent on the same questions and compare quality and cost.

4. Fix

• Typed briefs: objective, purpose, shared definitions, 'do not cover', output format, step limit.
• Typed reports: findings with sources and quotes, confidence, and a required 'not found'.
• Aggregate in code: merge, deduplicate, detect conflicts, carry gaps. The writer works only from that and must state conflicts.
• Private contexts; artifacts by reference.
• Reduce the number of agents to what the task's real structure needs, possibly one.

5. Prevent it recurring

• Evaluation cases with a planted conflict that must appear in the report as a conflict.
• A standing comparison against the single-agent baseline for quality and cost.
• Validation of every message between agents; one trace id across them.

1. Clarify

• Where is a run's state held between steps?
• How are approvals represented while waiting?
• Which actions repeated, and how do retries and resumes call them?

2. Likely causes, most probable first

• Run state is in worker memory, or in a cache that is cleared on restart.
• Waiting approvals are held as blocked processes rather than saved state.
• Workers are killed mid-step with no checkpoint, and the job is not returned to the queue.
• On resume the in-progress step runs again, and its write has no idempotency key.
• New code loads state saved by the old version and misreads it.

3. Confirm with evidence

• Kill a worker mid-task in a test environment and observe.
• For a duplicated action, check both calls in the audit log: same run, same step, different or missing keys.
• Inspect where the checkpoint is written relative to the tool call.

4. Fix

• Persist state to a durable store after every step; workers hold nothing between steps.
• Represent a waiting approval as a saved run with a status, woken by an event.
• Idempotency keys derived from run and action on every write; an action ledger in the state.
• Graceful shutdown: finish or checkpoint the current step, then return the job to the queue.
• Version the state; let in-flight runs finish on their original version, or migrate them.

5. Prevent it recurring

• A release gate: kill a worker at a random step and assert the run completes with each write performed once.
• A duplicate-delivery test for the queue.
• Alerts on runs with no progress for longer than expected.