LLM System Design · Module 8 of 17 · 7 concepts · about 11 min

Structured Output and Tool Calling

How to get an answer your code can use, not just read, and how to let a model take actions safely: schemas, validation, the tool-calling loop, permissions and limits.

At work: What turns a chatbot into a product that can look up an order, issue a refund or file a ticket, without those actions going wrong.

Structured output

What it is

Structured output means the model answers in a fixed, machine-readable format, usually JSON that matches a schema you supply. The schema states:

which fields must be present;
the type of each (text, number, true or false, one of a fixed list);
which are optional.

Most providers can now guarantee that the output matches the schema.

Why it matters

Free text is for people; code needs fields.

To route a ticket, your program needs category: "refund", not a sentence it has to interpret.
A schema also makes the model's job narrower, which makes it more reliable, and gives you something exact to validate.

In the real world: A support system classifies every incoming message into one of eight categories with a fixed schema, and routes it to the right queue without a person reading it first.

from pydantic import BaseModel
from typing import Literal

class Ticket(BaseModel):
    category: Literal["refund", "delivery", "payment", "other"]
    order_id: str | None
    urgent: bool
    summary: str

ticket = client.chat(messages=messages, response_format=Ticket)
ticket.category        # 'refund': a value your code can branch on

Asking for JSON in the prompt and parsing whatever comes back. Without a schema enforced by the provider, the model will sometimes add a sentence before the JSON, leave out a field or invent a category. Use the provider's schema feature where it exists.

Validation and repair

What it is

Even with a schema, the output is checked before it is used:

Shape: does it parse and match the schema?
Values: is the order ID a real order? Is the amount within limits?
Sense: do the parts agree with each other (do the line items add up to the total)?

When a check fails, the system repairs: it sends the error back to the model and asks again, a limited number of times.

Why it matters

A schema guarantees the shape, not the truth.

A well-formed answer can still name an order that does not exist.
Checks written in ordinary code are cheap, exact and never have an off day, so everything that can be checked that way should be, before a model's output is allowed to affect anything.

In the real world: An accounts team's invoice reader rejects any extraction whose line items do not sum to the stated total, and sends those few to a person; the rest go straight through.

def extract(invoice_text: str, attempts: int = 2) -> Invoice:
    messages = build_messages(invoice_text)
    for _ in range(attempts + 1):
        invoice = client.chat(messages=messages, response_format=Invoice)
        problem = check(invoice)          # plain code: totals add up, date valid
        if problem is None:
            return invoice
        messages.append(user_message(f"That was wrong: {problem}. Fix it."))
    raise ExtractionFailed("could not get a valid invoice")   # hand to a person

Retrying without limit, or without telling the model what was wrong. A bare retry often repeats the same mistake. Send the specific error, cap the attempts, and have a defined place for failures to go.

Typed tool calling

What it is

Tool calling lets a model ask your code to do something. You describe each tool: its name, what it does and its typed arguments. Then:

the model, instead of answering, returns a request to call a tool with specific arguments;
your code runs the tool (the model never runs anything itself);
the result is sent back to the model;
the model either calls another tool or writes the final answer.

Why it matters

It is how a model reaches live data and takes actions: looking up an order, checking stock, creating a ticket.

The typed arguments are structured output, so they can be validated before anything runs.
And because your code does the running, your code is where every permission check and limit lives.

In the real world: A delivery app's assistant answers 'where is my order' from a live lookup instead of guessing, and the lookup refuses any order that does not belong to the logged-in customer.

Visual example

Tool calling

The model asks; your code decides and acts.

The tool-calling loop

Describe toolsNames, purposes, typed arguments
Model requests a callA tool name and arguments
Your code validatesValid? Allowed for this user?
Your code runs itResult back to the model
Model continuesAnother call, or the answer

User:   Where is my order 88213?

Model:  call get_order(order_id="88213")          <- a request, not an answer
Code:   runs the lookup, checks the user owns this order
Result: {"status": "out_for_delivery", "eta_minutes": 12}

Model:  Your order is out for delivery and should arrive in
        about 12 minutes.

Believing the model executes the tool. It only produces text that names a tool and arguments. If your code runs whatever is named with whatever arguments are given, you have handed control of your systems to a text generator.

Designing tools a model can use

What it is

A model chooses tools from their descriptions alone, so tools are designed for it:

Few and distinct: a handful of clearly different tools beats dozens of overlapping ones.
Well described: say what the tool does, when to use it and when not to.
Narrow arguments: fixed choices where possible, not free text.
Useful errors: return a message the model can act on, such as 'no order with that ID'.

Why it matters

Most tool-calling failures are the model picking the wrong tool or filling an argument wrongly, and most of those come from vague descriptions and overlapping tools. A tool that returns a clear error lets the model correct itself; one that throws an exception ends the conversation.

In the real world: A team replaces fourteen database-shaped tools with five task-shaped ones and sees wrong-tool errors fall by more than half, with no change to the model.

Exposing your internal API as tools, one for one. Internal APIs are built for programmers who read documentation. A model does better with a few task-shaped tools ('find the customer's recent orders') than with twenty low-level ones it has to chain correctly.

Tool-level policy enforcement

What it is

Tool-level policy enforcement means the rules about what may be done are checked inside the tool's code, on every call:

who the user is comes from the logged-in session, never from the model;
the tool checks that this user may see this record or take this action;
business limits (a maximum refund, a daily cap) are enforced in code.

Why it matters

A model can be talked into things.

If the only rule against refunding ₹50,000 is a sentence in the prompt, a persistent or clever user will eventually get past it. A check in code cannot be argued with.
The prompt can explain the policy so the model behaves sensibly; the tool is what actually enforces it.

In the real world: A customer tells an airline's assistant that a supervisor approved a full refund; the model believes it and calls the refund tool, and the tool refuses because the amount exceeds what the fare rules allow.

def issue_refund(order_id: str, amount: int, *, session) -> dict:
    order = orders.get(order_id)
    if order is None or order.customer_id != session.customer_id:
        return {"error": "No such order for this customer."}
    if amount > min(order.total, MAX_AUTO_REFUND):
        return {"error": "Amount needs approval by an agent.", "escalate": True}
    return refunds.create(order_id, amount, idempotency_key=session.request_id)

Passing the customer ID as a tool argument the model fills in. The model can be persuaded to fill in someone else's. Identity must come from the session your server controls.

Idempotency for irreversible actions

What it is

An action is idempotent if doing it twice has the same effect as doing it once. For actions that cannot be undone (a refund, an email, a booking), each attempt carries an idempotency key:

a unique ID for this intended action;
the tool records the key the first time it succeeds;
a repeat with the same key returns the first result and does nothing more.

Why it matters

LLM systems retry constantly: a timeout, a failed validation, a model that calls the same tool twice.

Reading data twice is harmless. Refunding twice is not.
With idempotency keys, a retry is always safe, so the reliability techniques in later modules can be used without fear of double actions.

In the real world: A network glitch makes a refund call time out and retry; the customer receives one refund, not two, because the second attempt carried the same key.

Generating a fresh key on every retry. The key must identify the intended action, so it stays the same across retries of that action and changes only for a different one.

Bounded loops and human approval

What it is

A model that can call tools repeatedly is an agent, and its loop needs limits:

a maximum number of steps;
a token and time budget for the whole task;
a stop when it repeats the same call;
human approval before risky or irreversible actions.

Approval is usually tiered: reading is automatic, small actions are automatic, large ones wait for a person.

Why it matters

Without limits, a confused model can loop until the budget is gone, and a mistaken one can do real damage at machine speed.

Limits turn the worst case from 'unbounded' into 'a few wasted calls'.
Tiered approval puts a person exactly where an error would be expensive, and nowhere else.

In the real world: A support agent may look up orders and issue refunds under ₹500 on its own; anything larger is drafted and sent to a human agent with the full context attached.

Visual example

Tiered autonomy

The more it costs to be wrong, the more a person is involved.

Matching the level of oversight to the risk

kind of action	example	who decides
Read only	Look up an order	The model, automatically
Small and reversible	Add a note; refund under ₹500	The model, within coded limits
Large or irreversible	Refund ₹5,000; cancel a subscription	A person approves a draft
Never allowed	Change another customer's data	Blocked in the tool

Asking for approval on everything. Reviewers who are asked to approve a hundred trivial actions a day stop reading and click yes. Reserve approval for the few actions that warrant it.