Agentic AI · Module 23 of 41 · 11 concepts · about 17 min

Production Agent Architecture

The full system around an agent, layer by layer: gateway, orchestrator, reasoning engine, tool layer, memory layer and external systems, with the queues, databases, caches, identity, guardrails and observability that run through them.

At work: The diagram a senior engineer draws in a design review, and the one an interviewer expects on the whiteboard when the question begins 'design an agent that…'.

The reference architecture

What it is

A production agent system has six layers, top to bottom, and four concerns that cut across all of them:

API gateway: the front door. Authenticates, rate-limits and routes requests.
Agent orchestrator: runs the loop or workflow, owns state and limits.
Planner and reasoning engine: the model calls, behind a model gateway.
Tool layer: validates, authorises and executes tool calls.
Memory layer: run state, conversation history, long-term memory, document indexes.
External systems: the databases, APIs and services the tools act on.

Cutting across: identity and authorisation, guardrails, observability, and asynchronous infrastructure (queues and workers).

Why it matters

Layers give each concern one home.

A rule about who may call what lives in the tool layer, not scattered through prompts.
A change of model provider touches only the model gateway.
When something fails, the layers tell you where to look.
The model is one box in a system that is mostly ordinary, well-understood infrastructure.

In the real world: A design review rejects a proposal whose diagram has no state store and no tool layer; the revised one adds both and the question 'what happens on a crash?' has an answer.

Visual example

Reference architecture

Identity, guardrails and tracing accompany it through every layer.

A request's path through a production agent system

User and channelApp, chat, API or event
API gatewayAuthenticate, rate-limit, route
OrchestratorLoad state, run loop, save state
Reasoning engineContext; model via gateway
Tool layerValidate, permit, execute
Memory layerState, history, indexes
External systemsOrders, payments, email

Drawing 'LLM' in the middle with arrows to everything. That diagram has no place for identity, limits or state, which is where production systems succeed or fail.

API gateway, authentication and authorisation

What it is

The API gateway is the single entry point. It does four jobs before any model is called:

Authentication: establishes who the caller is, from a session or token.
Rate limiting: caps requests per user and per tenant.
Input checks: size limits, format, and a first screen for abuse.
Routing: sends the request to the right agent and version.

Authorisation, deciding what this caller may do, is then enforced deeper, in the tool layer, using the identity the gateway attached.

Why it matters

Identity must be established outside the model and carried unchanged to every tool.

If identity comes from the conversation, anyone can claim to be anyone.
Rate limits at the door protect both cost and downstream systems.
The agent acts on behalf of the user: tools should run with the user's rights, not with a powerful shared account.
In systems serving many customer organisations (multi-tenant), the tenant id is part of identity and filters every query.

In the real world: An enterprise assistant passes the employee's own access token to every tool; a question about salaries returns only what that employee could have seen in the HR system anyway.

def handle_request(request) -> Response:
    user = authenticate(request.headers["Authorization"])       # who is this? (never the model's job)
    if not rate_limiter.allow(user.id, tenant=user.tenant_id):
        return Response(429, "Too many requests")
    if len(request.body["message"]) > 8_000:
        return Response(413, "Message too long")

    context = RequestContext(user_id=user.id, tenant_id=user.tenant_id,
                             scopes=user.scopes, trace_id=new_trace_id())
    run_id = orchestrator.start_or_resume(request.body["conversation_id"], context,
                                          request.body["message"])
    return Response(202, {"run_id": run_id})                     # work continues asynchronously

A single service account for all tool calls, with the user's id passed as a parameter the model fills in. Every user then has the service account's power, one persuasive message away.

The agent orchestrator

What it is

The orchestrator is the code that runs the agent. It owns:

the loop or graph: which step runs next;
state: loading it, saving it after every step (module 6);
limits: steps, tokens, cost and time per run;
pauses: approvals and waits, and resuming from them;
failure handling: retries, and handing over when it cannot continue;
versioning: which prompt, tools and model this run uses.

Why it matters

It is where reliability is decided.

The orchestrator should be stateless: all state in a store, so any instance can continue any run.
For long tasks it should be durable: a crash loses at most one step. Engines built for this are called durable execution engines; graph frameworks with a database checkpointer provide the same property.
A run should finish on the version it started with, so a deployment mid-task does not change its rules.

In the real world: A support platform's orchestrator is forty lines around a graph with a Postgres checkpointer; deployments at noon no longer interrupt the two thousand conversations in progress.

Running the agent inside the web request that started it. A thirty-step task outlives any sensible HTTP timeout, and the user's closed browser tab kills the run. Start the run, return its id, and do the work in a worker.

The reasoning layer and model gateway

What it is

The reasoning layer is where models are called. Calls go through a model gateway, an internal service that every model request passes through. It provides:

one interface over several providers and models;
routing: which model handles which kind of step (a small one for classification, a large one for hard decisions);
fallback: a second model or provider when the first fails or is rate-limited;
retries with backoff, and timeouts;
caching of repeated prompts;
accounting: tokens and cost per run, user and feature;
keys and policy in one place.

Why it matters

Without a gateway, every team re-solves the same problems inconsistently.

A provider outage should mean a fallback, not an outage of yours.
Cost per task can only be capped if someone is counting.
Model upgrades become a configuration change tested against the evaluation set, not a code change in twenty places.
The planner, executor and critic of earlier modules are all just callers of this layer.

In the real world: When a provider had a two-hour incident, a company with a model gateway failed over automatically; a competitor with direct calls in every service was down for the duration.

Hard-coding a model name throughout the codebase. Models are retired and replaced on a schedule you do not control; one named in forty files is a migration project.

The tool layer

What it is

The tool layer stands between the model's requests and your systems. Every call passes through the same pipeline:

resolve the tool; reject unknown names;
validate inputs against the schema;
authorise: may this user do this, to this resource?
apply policy: caps, approvals, restricted actions;
execute, with a timeout, an idempotency key and, for code, a sandbox;
shape the result, and scan it if it came from outside;
record the call for the audit trail.

Why it matters

This is the layer that makes an agent safe to connect to real systems.

It is the only place where a rule cannot be argued with, because it is code.
Centralising it means a new tool gets validation, authorisation and logging by default.
Tools may be local functions, internal services or MCP servers; the pipeline is the same.
Secrets (API keys, database passwords) live here and never enter the model's context.

In the real world: A company's tool layer is a shared library; a team adding a new tool writes the function and its schema and inherits authorisation, idempotency, timeouts and audit logging.

def execute_tool(call, ctx: RequestContext, state) -> ToolResult:
    tool = REGISTRY.get(call.name)
    if tool is None:
        return error(f"Unknown tool. Available: {sorted(REGISTRY)}")
    if problem := tool.validate(call.input):
        return error(f"Invalid input: {problem}")
    if not authorise(ctx, tool, call.input):                       # the user's own rights
        return error("Not permitted for this user.")
    if approver := policy.needs_approval(tool, call.input, ctx):
        return pause_for_approval(state, call, approver)

    key = f"{state.run_id}:{state.step}:{call.name}"               # stable across retries
    with timeout(tool.timeout_seconds), span("tool", name=call.name):
        raw = tool.run(call.input, ctx, idempotency_key=key)       # secrets resolved inside
    audit.record(ctx, call, raw)
    return ok(shape(raw, max_tokens=800))

Letting the model see credentials 'so it can call the API'. Anything in the context can be repeated in an answer or extracted by a crafted input. The tool layer holds the key; the model holds only the tool's name.

The memory layer and its stores

What it is

The memory layer is several stores, each suited to one job:

Run state store (a relational database or durable-execution store): checkpoints of runs in progress.
Conversation store (relational): message history and summaries.
Long-term memory (relational plus a vector index): user profiles and remembered facts.
Vector database: indexes of documents for retrieval, and of memories.
Cache (an in-memory store such as Redis): repeated model responses, tool results, session data.
Object storage: large artifacts: files, reports, datasets.
Audit log (append-only): every action taken, by whom, on whose behalf.

Why it matters

One store cannot do all of these well.

State needs transactions and exact reads.
Retrieval needs similarity search.
Caches need speed and expiry.
Audit needs to be unalterable.
Each store also has its own access rule: a tenant's memories and documents must be filtered by tenant id in the store, never by the model.

In the real world: A team that kept checkpoints in Redis lost every paused approval when the cache restarted; state moved to Postgres and the cache went back to being a cache.

Visual example

Stores

Choose by what must be true of the data, not by what is already installed.

Which store holds what

data	store	must be
Run checkpoints	Relational database	Durable, transactional
Conversation history	Relational database	Ordered, per user
User profile and facts	Relational database	Exact, updatable, deletable
Documents and memories for search	Vector database	Searchable by meaning, filtered by tenant
Repeated responses and results	Cache	Fast, expiring
Files and large outputs	Object storage	Cheap, referenced by id
Actions taken	Append-only audit log	Complete, unalterable

Putting run state in a cache because it is fast. Caches evict. A checkpoint that disappears under memory pressure is a lost task.

Queues and asynchronous execution

What it is

A queue is a holding line of jobs; workers are processes that take jobs from it. Agents use queues because their work is slow and bursty:

the API accepts a request, puts a job on the queue and returns at once;
a worker picks it up and runs the agent;
progress and results reach the user by streaming, a webhook (a call back to the user's system) or polling;
separate queues isolate kinds of work: interactive chats ahead of overnight batch jobs.

Why it matters

A model call takes seconds and a task takes many calls.

Holding a web connection open for a five-minute task fails on timeouts and dropped connections.
A queue absorbs a burst instead of overloading the model provider's rate limit.
If a worker dies, the job returns to the queue and another worker resumes from the last checkpoint.
Workers can be added or removed as the queue grows or shrinks.

In the real world: A report-generation agent's API returns a job id in fifty milliseconds; the report arrives by webhook four minutes later, and a 9 a.m. burst of three hundred requests drains steadily instead of failing.

A queue with no limit on retries. A job that crashes its worker every time (a poison message) is retried for ever and blocks the line. Cap attempts and move failures to a dead-letter queue for a person to inspect.

Guardrails and observability across the layers

What it is

Two concerns touch every layer:

Guardrails (module 31): input checks at the gateway, policy checks in the tool layer, output checks before the reply, and limits in the orchestrator.
Observability (module 25): one trace per run, carrying the same id through gateway, orchestrator, model gateway and tool layer, with tokens, cost, latency and errors recorded at each step.

Why it matters

They work only if they are built into the layers, not bolted on.

A guardrail at just one layer has gaps; several thin layers cover each other.
A trace id created at the gateway and passed everywhere is what lets you reconstruct a run after the fact.
Together they answer the two questions every incident raises: what happened, and why was it allowed?

In the real world: During an incident a team pulls one trace id and sees the gateway request, nine model calls, six tool calls and the policy check that should have fired; the fix is identified in ten minutes.

Visual example

Cross-cutting

Read down a column to see one concern end to end.

Where each cross-cutting concern lives in each layer

layer	identity	guardrails	observability
API gateway	Authenticate; attach user and tenant	Rate limit; input size and abuse screen	Create the trace id
Orchestrator	Carry context with the run	Step, cost and time limits	A span per step
Reasoning	Per-user accounting	Prompt-level rules; output schema	Tokens, cost, latency per call
Tool layer	Authorise as the user	Policy, caps, approvals, sandbox	A span per tool; audit record
Memory layer	Filter by user and tenant	No sensitive data stored unneeded	Retrieval and cache metrics

Treating these as phase two. An agent launched without tracing cannot be debugged, and one launched without limits will, sooner or later, produce a large bill or a wrong action that nobody can explain.

Practise it: Design a Multi-Agent Incident Response System, Design a Self-Correcting Code Agent, Design a Tool-Calling Agent Over 500 APIs

Deployment, versions and rollout

What it is

An agent's behaviour is defined by more than code. A version is the combination of:

the prompts;
the tool definitions and their implementations;
the model and its settings;
the workflow or graph;
the guardrail rules.

A change to any one is a release, and is rolled out in stages: evaluation set first, then shadow mode (runs on real inputs with actions disabled), then a canary (a small share of traffic), then everyone, with a quick way back.

Why it matters

Any of these can change behaviour as much as a code change.

A new model version from the provider can alter results with no change on your side.
Staged rollout limits the damage of a bad change to a few users.
Recording the version on every trace lets you tie a shift in behaviour to the change that caused it.
Runs in progress should complete on their original version.

In the real world: A prompt change that passed the evaluation set raised the escalation rate in a 5 percent canary; it was rolled back in minutes, having affected a few dozen conversations instead of all of them.

Editing a production prompt directly 'because it is only text'. It is the most behaviour-changing artefact you have. It goes through review, evaluation and staged rollout like code.

Interview questions: production agent architecture

What it is

The questions interviewers ask about production agent architecture, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: The strongest system-design answers draw the tool layer as its own box with 'authorise, policy, idempotency' written inside it before the interviewer asks about safety.

Q: Draw the architecture of a production agent system.

User to API gateway (authentication, rate limits) to orchestrator (loop or graph, state, limits) to reasoning layer via a model gateway, to a tool layer (validate, authorise, policy, execute) acting on external systems, with a memory layer of state, history, long-term memory, vector index and cache. Queues and workers run it asynchronously; identity, guardrails and tracing cut across.

Q: Where is authorisation enforced?

In the tool layer, using identity established at the gateway and carried in the request context. Tools run with the user's rights. The model never supplies identity.

Q: Why a model gateway?

One place for provider abstraction, routing between models, fallback, retries, caching, cost accounting and key management. It makes outages survivable and model upgrades a configuration change.

Q: Why run agents asynchronously?

Tasks take many model calls and can run for minutes. A queue and workers avoid request timeouts, absorb bursts, respect provider rate limits, and let a failed worker's job be resumed by another.

Q: Which data stores does an agent system need?

A durable store for run checkpoints, a relational store for history and profiles, a vector index for retrieval, a cache for repeated work, object storage for artifacts, and an append-only audit log.

Q: What makes an orchestrator production-grade?

Statelessness with saved checkpoints, limits on steps, cost and time, idempotent tool calls, pause and resume, retry and hand-over paths, and runs pinned to the version they started on.

Q: How do you roll out a change?

Version prompts, tools, model and workflow together; run the evaluation set; shadow on real traffic with actions disabled; canary to a small share; then full rollout with rollback ready and the version stamped on every trace.

Q: Where do secrets live?

In the tool layer, resolved at execution from a secrets manager. Never in prompts, tool results or anything the model reads.

A diagram where the LLM connects directly to the database.

Exercise: design a production architecture

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A platform team keeps this one-page diagram for every agent in production; new engineers read it before they read any code.

Your task. Produce a one-page architecture for the Kirana Kart support agent serving 100,000 conversations a day, and defend it against five failure questions.

Steps

Draw the six layers and label what runs in each for this system.
List every data store, what it holds, and why that kind of store.
Mark where identity is established and every place it is checked.
Mark where each limit lives: per request, per run, per user, per tenant.
Show the asynchronous path: what the API returns immediately and how the customer gets the reply.
Write one paragraph each answering: a worker crashes mid-refund; the model provider is down; a prompt change goes wrong; one customer sends a thousand messages; an auditor asks why a refund was issued.

You are done when

No arrow connects the model directly to an external system.
Each of the five failure questions is answered by pointing at a specific box.
Every store has a stated reason, and state is not in a cache.
A colleague can find where a ₹500 cap is enforced within ten seconds.

Stretch. Add a second tenant with stricter rules and show what changes in each layer.