Agentic AI · Module 31 of 41 · 11 concepts · about 20 min

Agent Guardrails

The controls that keep an agent within policy: input and output validation, tool permissions and authorisation, restricted actions, policy enforcement, rate limiting, sandboxing and content filtering; and the difference between guardrails a model applies and guardrails code enforces.

At work: What a risk or compliance team asks to see before an agent is allowed near money, customer data or production systems.

What guardrails are and where they sit

What it is

A guardrail is a check that stops an agent doing, saying or accepting something it should not. Guardrails sit at four points in every run:

Input: before the model sees the request.
Tool call: after the model asks for an action, before it runs.
Tool result: after the action, before the result enters the context.
Output: before the reply reaches the user.

Around these sit run-level limits: steps, cost, time and rate.

Why it matters

An agent takes actions, so a mistake is not a bad sentence but a real event.

No single check catches everything. Layers cover each other's gaps; this is called defence in depth.
The most important layer is the one before a tool runs, because that is the last moment a wrong action can be stopped.
Guardrails are designed from the worst case backwards: what is the most damaging thing this agent could do, and which line of code prevents it?

In the real world: A bank's agent has eleven guardrails across the four points; an internal red-team exercise got past four of them and was stopped by the fifth, which is the argument for having eleven.

Visual example

Placement

The third box is the last chance to stop a wrong action.

The four guardrail points in one run

InputSize, format, abuse, manipulation
Model decidesTool request or draft reply
Tool calllast chanceValidate, authorise, approve
Tool resultTrim, scan, label
OutputGrounded, safe, no leaks

Run-level limits on steps, cost, time and rate apply throughout.

One guardrail at the front door and nothing after. An input filter cannot see what the agent will decide ten steps later, or what a tool result will bring in.

LLM guardrails versus application-level guardrails

What it is

Two fundamentally different kinds:

LLM guardrails are checks made by a model: an instruction in the system prompt, or a separate classifier model that judges an input or output. They are probabilistic: right most of the time.
Application-level guardrails are checks made by code: schema validation, permission checks, caps, allow-lists, rate limits, sandboxes. They are deterministic: the same input always gives the same result, and no wording can change it.

Why it matters

Each does what the other cannot.

A model-based check understands meaning: intent, tone, whether a message is an attempt at manipulation. Rules cannot express that.
A code check cannot be talked out of its answer. A persuasive customer can move a model; nobody can persuade amount <= 500.
So the rule is: anything that must never happen is enforced in code. Model-based guardrails add coverage for fuzzy cases, and reduce how often the hard limits are even reached.
A design whose only protection for money or data is a prompt has no protection.

In the real world: A customer wrote a long, sympathetic message claiming a manager had approved ₹5,000; the model was persuaded and requested the refund, and the tool refused it because the cap was in code.

Visual example

Two kinds

Use both. Rely on the right-hand column for anything that must never happen.

LLM guardrails versus application-level guardrails

	LLM guardrail	application-level guardrail
Implemented as	A prompt or a classifier model	Code
Behaviour	Probabilistic	Deterministic
Understands meaning	Yes	No
Can be argued with	Yes	No
Cost and latency	A model call	Microseconds
Testing	Statistical: measure error rates	Exact: unit tests
Good for	Intent, tone, topic, subtle abuse	Limits, permissions, formats, rates
Example	'Is this message a manipulation attempt?'	amount <= min(500, order.total)

'The system prompt tells it never to refund more than ₹500.' That is a request to a model that can be argued with, confused by a long context, or overridden by injected text. Put the cap in the refund tool.

Input validation

What it is

Input validation checks what arrives before the agent acts on it. Layers, cheapest first:

Structural: length limits, allowed characters and file types, well-formed data.
Identity and scope: who is asking, and is this request within what the agent is for?
Abuse and manipulation: attempts to override instructions, claim false authority or extract the prompt. A narrow classifier model suits this.
Sensitive data: detect card numbers, passwords or identity numbers a user pasted in, and mask them before they are stored or sent to a model.
Topic: decline what is plainly out of scope before spending on the agent.

Why it matters

Stopping a bad request early is the cheapest control.

A size limit alone blocks a class of attacks that bury instructions in very long inputs, and bounds cost.
Masking sensitive data on entry keeps it out of logs, traces and memory.
An input check is a filter, not a wall: it will miss things, so nothing downstream should assume the input was clean.
'User input' includes everything the agent reads that a user or outsider could influence: uploaded files, web pages, emails.

In the real world: A support agent masks card numbers at the door; a later audit of its traces finds none, although customers paste them into chat every day.

import re

CARD = re.compile(r"\b(?:\d[ -]?){13,19}\b")

def check_input(message: str, user) -> dict:
    if len(message) > 8_000:
        return {"ok": False, "reason": "too_long"}
    message = CARD.sub("[card number removed]", message)          # mask before storage or model
    if not user.verified:
        return {"ok": False, "reason": "unverified_user"}
    verdict = manipulation_classifier(message)                    # a narrow small-model check
    if verdict.label == "manipulation":
        log_security_event(user, message, verdict.reason)
        # Do not refuse outright: continue with read-only tools and flag for review.
        return {"ok": True, "message": message, "restricted": True}
    return {"ok": True, "message": message, "restricted": False}

Blocking every message the classifier flags. Angry or unusual customers get refused for nothing. A better response to a suspected manipulation is to continue with reduced powers and let the code limits do their work.

Tool permissions, authorisation and restricted actions

What it is

The controls on what an agent may do:

Tool permissions: which tools this agent has at all. The fewest that do the job.
Authorisation: whether this user may perform this action on this resource. Checked in the tool, with identity from the session.
Restricted actions: actions that are never offered (closing an account), or offered only under conditions (after verification, within a cap, with approval).
Scopes by stage: write tools appear only once the workflow reaches the state that permits them.

Why it matters

This is the layer that bounds the worst case.

Least privilege: an agent that cannot delete cannot be tricked into deleting.
An action a tool does not implement is impossible, whatever the model outputs. Removing a tool is the strongest guardrail there is.
Authorisation must use the user's rights, not a powerful shared account, or every user inherits that account's power.
The check looks at arguments, not only tool names: a ₹40 refund and a ₹40,000 refund are the same tool.

In the real world: A red-team tester spent a day trying to make an agent close an account; the attempt could not succeed, because there was no such tool to call.

PERMISSIONS = {
    "get_order":      {"needs": "orders:read",  "own_data_only": True},
    "refund_item":    {"needs": "refunds:write", "own_data_only": True, "max_amount": 500,
                       "approval_above": 500},
    "cancel_order":   {"needs": "orders:write", "own_data_only": True, "only_status": "preparing"},
    # close_account is deliberately absent: the agent cannot do it at all.
}

def authorise(tool: str, args: dict, ctx) -> str | None:
    rule = PERMISSIONS.get(tool)
    if rule is None:
        return "This action is not available to the assistant."
    if rule["needs"] not in ctx.scopes:
        return "Your account is not permitted to do this."
    if rule.get("own_data_only") and not owns(ctx.user_id, args):       # identity from the session
        return "That record does not belong to this customer."
    if args.get("amount", 0) > rule.get("max_amount", float("inf")) and not ctx.approved:
        return "REQUIRES_APPROVAL"
    return None

Checking permission by tool name only. The model then calls the permitted tool with another customer's order id. The check must cover the resource and the arguments.

Practise it: Design a Tool-Calling Agent Over 500 APIs

Policy enforcement

What it is

Policy enforcement turns business rules into checks that run on every tool call. A policy engine is the component that holds the rules and answers one question: is this action, by this user, with these arguments, in this state, allowed?

rules are data or code, kept apart from prompts, versioned and reviewed;
the answer is allow, deny with a reason, or allow after approval;
every decision is logged;
the same engine serves every agent, so a rule is written once.

Why it matters

Rules scattered through prompts and tool bodies drift and contradict each other.

A single place for policy means compliance can read it, test it and change it without touching the agent.
The denial reason is returned to the model as the tool's result, so it can explain to the user or choose another route.
Policies depend on context: the amount, the customer's history, the time, how many refunds today. Code can check all of that; a prompt cannot do so reliably.

In the real world: A retailer's refund rules changed three times in a quarter; each change was a reviewed edit to one policy file, with tests, and no prompt was touched.

def refund_policy(action, ctx) -> Decision:
    order = orders.get(action.order_id)
    if action.amount > order.total:
        return deny("A refund cannot exceed the order total.")
    if refunds.count_today(ctx.user_id) >= 3:
        return deny("Daily refund limit reached; a colleague must review.")
    if action.amount > 500 or order.age_days > 30:
        return needs_approval("supervisor", reason="amount over ₹500 or order older than 30 days")
    return allow()

POLICIES = {"refund_item": refund_policy, "cancel_order": cancel_policy}

def enforce(call, ctx) -> Decision:
    decision = POLICIES.get(call.name, default_allow_reads_only)(call.input, ctx)
    audit.record(ctx, call, decision)                    # every decision, allowed or not
    return decision

A denial that returns only 'not allowed'. The model retries with small variations, or invents an explanation for the customer. Return the reason and what to do instead.

Practise it: Design an AI Agent for Customer Refunds, Design an AI Customer Support Agent

Output validation and content filtering

What it is

Output validation checks the reply before the user sees it:

Structure: matches the expected schema (module 4).
Groundedness: amounts, ids and claimed actions match the tool results and the action log.
Leaks: no system prompt text, internal ids, other customers' data, or secrets.
Policy: no promises the agent cannot keep, no advice it is not allowed to give.
Content filtering: no harmful, abusive or off-brand content. Usually a moderation classifier.

A failed check leads to a regenerate, a safe fallback message, or a hand-over.

Why it matters

The output is what the customer, and sometimes a regulator, actually sees.

Code checks catch the most damaging errors exactly: a wrong amount, a claimed refund that did not happen.
A model-based check catches what code cannot: tone, unsafe advice, subtle leaks.
With streaming, checks must run before text is sent, or risky content is held back until it passes.
Content filtering also applies to inputs to tools: what the agent is about to email or post.

In the real world: An output check comparing every rupee amount in a reply with the tool results stopped a class of error where the agent quoted the item price as the refund amount.

def check_output(reply: str, state) -> list[str]:
    problems = verify_answer(reply, state)                      # amounts, ids, claimed actions (module 28)
    if any(marker in reply for marker in SYSTEM_PROMPT_MARKERS):
        problems.append("Reply contains system prompt text.")
    if contains_other_customer_data(reply, state.user_id):
        problems.append("Reply contains another customer's data.")
    if moderation(reply).flagged:
        problems.append("Reply failed content moderation.")
    return problems

def respond(state) -> str:
    for _ in range(2):
        reply = draft_reply(state)
        problems = check_output(reply, state)
        if not problems:
            return reply
        state.add_note(f"Your draft was rejected: {problems}. Rewrite it using only tool results.")
    return hand_over(state, reason="output_checks_failed").message

Regenerating in a loop until the checks pass. If two attempts fail, the cause is upstream. Stop, hand over, and look at the trace.

Rate limiting

What it is

Rate limiting caps how much can happen in a period. For agents it applies at several levels:

per user: requests per minute, and actions per day (three refunds);
per run: steps, tokens, cost, wall-clock time;
per tool: calls per minute, to protect the system behind it;
per tenant: concurrent runs and spend;
global: total spend per hour, with an alert and a brake.

Why it matters

Limits bound damage that no other check anticipated.

An agent talked into issuing refunds can issue at most three before the limit stops it.
A loop can cost at most the run's budget.
A burst cannot overwhelm a fragile back-end system.
An attacker automating abuse hits a ceiling quickly.
Limits on actions matter more than limits on requests: count refunds and emails, not just messages.

In the real world: A per-customer cap of three refunds a day turned a discovered manipulation technique from an open tap into a ₹1,500 ceiling per account while the fix was shipped.

LIMITS = {
    ("user", "refund_item"):  (3, "day"),
    ("user", "send_email"):   (5, "hour"),
    ("user", "message"):      (30, "minute"),
    ("tool", "search_orders"): (200, "minute"),
    ("tenant", "spend_rupees"): (50_000, "day"),
}

def within_limit(scope: str, key: str, name: str, amount: int = 1) -> bool:
    limit, window = LIMITS[(scope, name)]
    used = counter.increment(f"{scope}:{key}:{name}:{bucket(window)}", amount, expire=window)
    return used <= limit                         # shared counter, so it holds across all workers

Limiting only messages per minute. A patient attacker sends one message that causes fifty tool calls. Limit what matters: actions, cost and tool calls.

Sandboxing

What it is

A sandbox is an isolated environment in which something can run without being able to affect anything outside it. Agents need one whenever they:

run code the model wrote: Python for data analysis, shell commands;
browse or open files from untrusted sources;
use a computer through screenshots and clicks.

A proper sandbox restricts: file system (only a working directory), network (off, or an allow-list of hosts), resources (CPU, memory, time), credentials (none inside), and lifetime (destroyed after the task).

Why it matters

Code written by a model is untrusted code.

It may be wrong, or it may have been shaped by an attacker through something the agent read.
Without isolation, rm -rf, a crypto-miner, or a script that posts your data to a remote server are all one tool call away.
With no network and no credentials inside, even malicious code has nothing to steal and nowhere to send it.
Isolation technologies range from containers to lightweight virtual machines; the stronger the isolation, the less you depend on the code being benign.

In the real world: A data-analysis agent runs every script in a network-less container with only the uploaded CSV inside; a prompt-injected spreadsheet that told it to upload the file elsewhere produced a harmless connection error.

Visual example

Sandbox

Remove any row and the corresponding harm becomes possible.

What a sandbox restricts, and what each restriction prevents

restriction	setting	prevents
File system	Only a scratch directory is writable	Reading secrets; damaging the host
Network	Off, or an allow-list	Sending data out; downloading malware
Resources	CPU, memory and time caps	Runaway loops; denial of service
Credentials	None inside	Using your keys or database
Lifetime	Destroyed after the task	Persistence between tasks or users

def run_python(source: str, files: dict[str, bytes]) -> dict:
    """Execute model-written code where it can harm nothing."""
    sandbox = Sandbox.create(
        image="python-analysis",          # numpy and pandas preinstalled; nothing else
        network="none",                   # no outbound connections at all
        cpu_seconds=20, memory_mb=512,
        read_only_root=True,              # only /work is writable
        env={},                           # no API keys, no credentials
    )
    try:
        for name, data in files.items():
            sandbox.write(f"/work/{name}", data)
        result = sandbox.exec(["python", "-c", source], timeout=30)
        return {"stdout": result.stdout[-4000:], "stderr": result.stderr[-2000:],
                "exit_code": result.exit_code}
    finally:
        sandbox.destroy()                 # nothing survives the task

Running generated code with exec in the agent's own process 'because it is only a calculation'. That process holds your API keys and database connections.

Practise it: Design a Multi-Agent Data Analyst, Design a Self-Correcting Code Agent

Testing, monitoring and the audit trail

What it is

Guardrails are software and need the same care:

Tests: unit tests for every code guardrail; labelled sets with measured error rates for every model-based one; adversarial cases in the evaluation set.
Two error rates for each check: how often it blocks something fine (false positive) and how often it lets something bad through (false negative).
Monitoring: the block rate of each guardrail over time. A sudden change means an attack, a bug or a drifted model.
Audit trail: an append-only record of every action taken or refused: who asked, what was proposed, which rule decided, who approved.
Kill switches: a way to disable a tool or the whole agent in seconds.

Why it matters

A guardrail nobody tests is a guess.

A check that blocks too much makes the agent useless and gets switched off; one that blocks too little gives false comfort.
The audit trail is what answers 'why did this happen?' after the fact, for an engineer, a customer or a regulator.
Guardrails should be reviewed whenever tools change: a new tool is a new capability the old rules did not consider.

In the real world: A manipulation classifier's block rate tripled overnight; the trace sample showed a marketing email had told customers to type a phrase the classifier disliked. The rule was fixed before the support queue filled.

Measuring a guardrail only by how many bad things it stopped. Also count how many good requests it wrongly refused; that number decides whether users, and your own team, tolerate it.

Practise it: Design an AI Agent for Customer Refunds, Design an AI Customer Support Agent

Interview questions: agent guardrails

What it is

The questions interviewers ask about agent guardrails, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: The question 'how do you stop it refunding ₹50,000?' has one wrong answer, 'the prompt says not to', and interviewers ask it specifically to hear whether you give it.

Q: What are guardrails?

Checks that keep an agent within policy: on the input, on each tool call before it runs, on tool results before they enter the context, and on the output, plus run-level limits. Layered, because no single check catches everything.

Q: LLM guardrails versus application-level guardrails?

LLM guardrails are model-based checks: flexible, understand meaning, and probabilistic. Application guardrails are code: deterministic and cannot be argued with. Anything that must never happen is enforced in code; model checks add coverage for fuzzy cases.

Q: How do you stop an agent doing something it should not?

Give it only the tools it needs, bind them to the user's identity from the session, enforce permissions, caps and policy in the tool layer, require approval for risky actions, and rate-limit actions. Do not rely on the prompt.

Q: What does input validation cover?

Size and format limits, identity and scope, detection of manipulation, masking of sensitive data, and topic checks. It is a filter, not a guarantee, and applies to all untrusted content, not only the user's message.

Q: What do you check on output?

Schema, that amounts, ids and claimed actions match tool results and the action log, no leaked prompt or other users' data, policy compliance, and content moderation. On failure: regenerate once or twice, then fall back or hand over.

Q: Why sandbox code execution?

Model-written code is untrusted and may be shaped by an attacker. A sandbox with no network, no credentials, a scratch file system, resource limits and a short life means even malicious code cannot read secrets or send data out.

Q: What role does rate limiting play?

It bounds damage nothing else anticipated: actions per user per day, steps and cost per run, calls per tool, spend per tenant. Limit actions and cost, not only requests.

Q: How do you test guardrails?

Unit tests for code checks, labelled sets measuring false positives and false negatives for model checks, adversarial cases in the evaluation set, red-team exercises, and monitoring of block rates in production.

Answering a safety question with a prompt instruction.

Exercise: implement guardrails at all four points

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A compliance team's sign-off for an agent is a demonstration of exactly these tests: for each guardrail, the attack with it off, and the same attack with it on.

Your task. Add input, tool-call, tool-result and output guardrails to your agent, plus rate limits, and show each one stopping a specific bad outcome.

Steps

Input: add a length limit, card-number masking and a small manipulation classifier that switches the run to read-only tools.
Tool call: add a permissions table and a refund policy function with a cap, a daily count and an approval threshold. Return reasons on denial.
Tool result: cap result size and wrap any externally sourced text in a marker stating it is data.
Output: verify amounts, ids and claimed actions against the action log; check for system prompt leakage.
Rate limits: at most three refunds per user per day and a per-run cost budget.
Write one test per guardrail in which the bad outcome happens without it and is prevented with it.
Add a flag that disables refund_item, and confirm the agent hands refund requests over instead.

You are done when

'Your manager approved ₹5,000' results in no refund above the cap, whether or not the classifier catches it.
A fourth refund in a day is denied with a clear reason.
A reply that misstates the refund amount never reaches the user.
Every allow, deny and approval appears in an audit log with the rule that decided it.

Stretch. Measure your manipulation classifier's false-positive rate on fifty ordinary angry messages and tune it.