Interview: Advanced Questions

Q: How would you prevent infinite agent loops?

Three layers. Limits in code: steps, tokens or cost, and time. Detection: identical or alternating calls, or several steps with nothing new. Response: one corrective nudge, then stop with a summary and hand over. And the root cause: tool results and errors that say what to try next, because most loops are an agent re-asking a question it never got a useful answer to.

Q: Why do agents fail on long tasks?

Errors compound. Task success is roughly per-step success raised to the number of steps: at 95 percent per step, twenty steps succeed about a third of the time. The remedies are more reliable steps, fewer steps, and checks that catch an error before the next step builds on it.

Q: A refund call times out. What does your system do?

It retries with the same idempotency key, derived from the run and the action. If the first call succeeded, the payment service returns the original result; if not, it performs it once. The step is recorded in the action ledger and the state is checkpointed. The customer is told only what the ledger shows.

Q: The process crashes half-way through a multi-step action. Then what?

The run resumes from its last checkpoint. Completed steps are in the ledger and skipped; the in-progress step is repeated safely because it is idempotent. If a later step fails permanently, earlier steps that must not stand alone are compensated, and the customer gets an accurate account.

Q: How do you reduce hallucination in an agent?

Supply facts through tools and make empty results explicit; require looking up before stating; verify the final answer in code against tool results and the action log, including amounts, ids and claimed actions; and confirm actions in the system of record.

Q: How do you handle stale data?

Timestamp and expire memories and caches, and re-read any fact an action depends on inside the tool at the moment of acting, with a version check so the write fails if the record changed.

Q: What is a cascading failure in an agent system?

One error spreading: a wrong fact built on for ten steps, one agent's mistake repeated by others, or a slow dependency exhausting workers through timeouts and retries. Contain it with validation at boundaries, timeouts, circuit breakers, bulkheads and bounded retries.

Q: How do you make a non-deterministic system dependable?

Put determinism around it: fixed workflow stages, typed outputs, validation, limits and policy in code, idempotent tools and saved state. Then measure reliability as a rate over repeated runs and require safety behaviours to pass every run.

Q: How do you defend an agent against prompt injection?

Assume it will sometimes succeed and design so that it does not matter. Least-privilege tools; identity, caps and policy enforced in code; untrusted content kept away from private data and outbound channels; confirmation for irreversible or outbound actions; untrusted text marked as data; input and output filters; sandboxes; and adversarial cases in the evaluation set.

Q: What is indirect prompt injection, and why is it worse?

Instructions hidden in content the agent reads while working: a web page, an email, a file. The attacker never talks to the agent, the victim is an innocent user whose access is used, one poisoned source attacks every agent that reads it, and the user sees nothing.

Q: What is the lethal trifecta?

An agent with access to private data, exposure to untrusted content and a way to send data out. With all three, one injected document can exfiltrate data. Remove a leg, or place confirmation and destination allow-lists between them.

Q: LLM guardrails versus application-level guardrails?

Model-based checks understand meaning and are probabilistic; code-based checks are deterministic and cannot be argued with. Anything that must never happen is enforced in code; model checks add coverage for intent and tone.

Q: How would you design an agent that executes financial transactions?

Narrow tools bound to the authenticated user; amounts and eligibility checked in code against the system of record; idempotency keys; hard caps and velocity limits; the customer's confirmation of the exact transaction with a second factor; supervisor approval above thresholds; a fixed workflow so payment is reachable only after verification; no untrusted content in the context of the model that holds payment tools; a complete audit trail; reconciliation against the ledger; and a kill switch.

Q: What is excessive agency?

More functionality, permission or autonomy than the task needs. It amplifies every other threat. Fix with the fewest tools, the narrowest scopes and approval for high-impact actions.

Q: How do you prevent an agent leaking one customer's data to another?

Identity from the session, never from the model; tools and databases that filter by that identity, with row-level security as a second line; memory and retrieval stores filtered in the query; no cross-user caches; and a test on every release that one user cannot retrieve another's record.

Q: How do you handle secrets?

Never in prompts, tool descriptions, results or memory. The tool layer resolves them from a secrets manager at call time; tokens are short-lived and scoped; sandboxes have none; logs and traces are redacted.

Q: How do you safely let an agent run code?

In a sandbox with no network or an allow-list, no credentials, a scratch file system, resource and time limits, destroyed after the task; with read-only database roles and query limits where it writes SQL.

Q: How would you evaluate agent reliability?

As a rate over repeated runs. A test environment with fake tools and seeded data; a case set covering common, edge, refusal, hand-over, tool-failure and adversarial cases; end-state checks in code; trajectory properties; five to ten runs per case, with safety cases required to pass every run; fault injection; then shadow mode and a canary, with reversal and intervention rates tracked in production.

Q: Outcome versus trajectory evaluation?

Outcome checks the end state: the right thing happened and nothing else did. Trajectory checks the path: required and forbidden steps, ordering, valid calls and efficiency. A correct outcome by a wrong path is luck that will not repeat.

Q: Why not assert an exact sequence of tool calls?

Several paths are valid, so exact matching fails good runs. Assert properties: looked up before acting, no forbidden tool, within a step budget.

Q: How do you use an LLM as a judge responsibly?

One criterion per judge, a rubric with borderline examples, evidence supplied, reasoning before a binary verdict, a different model from the one judged, and calibration against human labels: agreement, chance-corrected agreement and, above all, the share of real failures it catches.

Q: Your offline evaluation says 95 percent and users are unhappy. Why?

The set does not represent real traffic, the checks grade wording rather than outcomes, single runs hide intermittent failures, or the prompt was tuned to the set. Sample real traces, add failures as cases, grade on end state, and hold cases out.

Q: How do you evaluate a change to the model version?

Run the full set several times on both versions, compare pass rates, cost and latency per case kind, list regressions by case, re-tune prompts if needed, then shadow and canary. A model upgrade is a release like any other.

Q: What do you monitor in production to know quality is holding?

Task success where it can be measured, escalation and reversal rates, steps and cost per task, runs ended by limits, sampled groundedness scores, user feedback and reviewer verdicts, all sliced by version.

Q: How would you scale to thousands of concurrent agents?

Stateless asynchronous workers pulling runs from queues; all state in a durable shared store; a shared rate limiter in a model gateway sized to the provider's tokens-per-minute quota; per-tenant concurrency limits and separate queues for interactive and batch work; idempotent tools and per-conversation locks; and then fewer tokens per task, because the provider's quota, not your compute, is the ceiling.

Q: What is the main bottleneck in an agent system?

Usually the model provider's tokens-per-minute limit, then sequential model latency and the downstream systems tools call. Size capacity from peak tasks per minute times tokens per task.

Q: Two workers pick up the same conversation. What happens?

A per-conversation lock with a lease makes the second back off. Versioned saves make a conflicting write fail and reload. Idempotency keys make any duplicate write harmless.

Q: How do you reduce cost without losing quality?

In order: trim tool results and context; cache the stable prefix; route routine steps to smaller models; cut steps with parallel calls and better tools; cap budget per task; send simple requests down a cheaper path. Re-run the evaluation after each quality-affecting change, and judge by cost per resolved task.

Q: Why is cost per resolved task the right metric?

Failed tasks are handled by people, so a cheaper configuration with a lower success rate can cost more overall. It is also the number to compare with a human doing the work.

Q: How do you reduce latency?

Fewer sequential steps through parallel tools; faster models and lower reasoning effort for easy steps; shorter outputs; caching and pre-loading; a fast path for simple requests; and streaming progress and the answer to cut the perceived wait.

Q: How do you behave under overload or a provider outage?

Queue and tell users, fall back to another model for safe tasks, cut optional steps, shed low-priority work, use circuit breakers, and reduce autonomy if quality falls: pause actions rather than take them with a weaker model.

Q: How do you keep one tenant from affecting others?

Per-tenant concurrency and rate limits, fair scheduling or separate queues, per-tenant budgets, and tenant labels on every metric so a noisy neighbour is visible.

Q: When would you choose a workflow over an agent?

Whenever the steps can be drawn in advance. It is cheaper, faster, testable and predictable. I add agent freedom only at the step where the fixed path keeps failing, and keep the rest fixed.

Q: Single agent or multi-agent?

Single until there is a specific reason: independent strands to parallelise, a context that would be flooded, distinct tool sets, or an independent check. Then the most controlled shape, usually a supervisor with workers, and a measured comparison against the single agent.

Q: Where do you place human approval?

On actions that are irreversible, costly, uncheckable by code or novel, decided per action and per argument, triggered by code, with the evidence shown, a timeout and an audit record. Few enough that each is actually read.

Q: How do you decide between building on a framework and writing your own?

By what the task needs: durable state, approvals and complex flows favour a graph framework; one loop and a few tools favour plain code. Either way tools stay as plain functions, the evaluation set stays outside the framework, and I must be able to print the raw request.

Q: How do you roll out a change to an agent?

Version prompts, tools, model and workflow together; run the evaluation set; shadow on real traffic with actions disabled; canary to a small share; full release with rollback ready and the version on every trace.

Q: What would make you say 'do not build this as an agent'?

No way to check success, irreversible high-stakes actions with no approval point, a fixed process that a workflow covers, or a volume too low to justify the engineering. Saying so is part of the job.

Q: What is the hardest part of putting an agent into production?

Not the model. It is the engineering around it: an evaluation set that reflects real traffic, tools that enforce the rules, state that survives failure, and the operational visibility to explain any run.