Agentic AI · Module 26 of 41 · 9 concepts · about 16 min

Agent Cost Optimisation

Where an agent's money goes and the levers that reduce it: trimming context, caching, routing work to smaller models, limiting loops, compressing prompts and memory, caching tool results and running steps in parallel; with a worked example that applies them in order.

At work: The work that turns a pilot the finance team is nervous about into a product with a known cost per resolved task that falls as volume grows.

Where the money goes

What it is

An agent's bill is mostly model tokens. Per task:

cost = (input tokens × input price) + (output tokens × output price), summed over every model call;
input tokens dominate, because each step resends the whole history, so the same text is paid for many times;
output tokens are fewer but priced several times higher;
tool and infrastructure costs are usually small beside these.

So the levers are: send fewer tokens, pay less per token, and make fewer calls.

Why it matters

You cannot reduce what you have not measured.

A ten-step task does not cost ten times one step: later steps carry everything from earlier ones.
The fixed part (system prompt and tool definitions) is paid on every step, so it is worth caching.
What each step adds (mostly tool results) is paid again on every later step, so it is worth trimming.
The numbers below use round example prices; your own will differ but the ratios hold.

In the real world: A team's estimate for its research agent was out by a factor of four because it priced ten steps at the size of the first; the per-step token chart corrected the budget before launch.

Visual example

Baseline

The context starts at 3,000 tokens and each step adds 1,500. Medium model, example prices.

What one 10-step task costs before any optimisation

	value
Input at step 1	3,000
Input at step 10	16,500
Total input over the task	97,500
Total output	2,000
Cost of one task	₹26.88
Cost of 100,000 tasks	₹26.9 lakh

Input is 91 percent of the cost. That is where to look first.

Estimating cost as 'steps × price of one call'. That misses the growth. Sum the input of every step, as the table does.

Context trimming and prompt compression

What it is

The first lever is to send less:

Trim tool results at the source: return the fields the agent needs, in plain words, with a size cap.
Compress prompts: cut instructions that no test depends on; shorten tool descriptions without losing the 'when to use' line; offer fewer tools.
Clear stale results: drop old tool outputs that later steps replaced.
Summarise memory and history: compact old turns (module 12).

Why it matters

Trimming what each step adds has a compounding effect.

A token saved at step 2 is saved again at steps 3 to 10.
Tool results are usually the largest and least examined part of the context.
Shorter contexts are also faster and often more accurate.
This lever costs nothing in quality when done well, so it comes first.

In the real world: Trimming three tool results was a two-hour change that cut a support agent's cost per task by more than half; no prompt or model changed.

def shape_order(order: dict) -> str:
    """3,000 tokens of raw JSON become about 60 tokens the model can use."""
    return (f"Order {order['id']}: {len(order['items'])} items, total ₹{order['total']}, "
            f"fee ₹{order['fee']}, promised {order['promised']}, delivered {order['delivered']} "
            f"({order['late_minutes']} min late), status {order['status']}.")

def cap(text: str, max_tokens: int = 800) -> str:
    """Never let one tool result flood the context."""
    if count_tokens(text) <= max_tokens:
        return text
    return truncate(text, max_tokens) + "\n[truncated; ask for a specific part if needed]"

Compressing a prompt by deleting the examples and reasons that made it work. Compress against your test set: a cut that lowers the score is not a saving.

Caching: prompts, responses and tool results

What it is

Three different caches:

Prompt caching: the provider stores the unchanged start of your context (system prompt, tool definitions, early history) and charges a fraction, around a tenth, to re-read it on later calls.
Response caching: store a model's answer to a request and reuse it for an identical one. Semantic caching reuses it for a request with the same meaning.
Tool result caching: store a tool's result for a short time, so repeated look-ups of the same policy or product do not call the back end, or re-enter the context differently, each time.

Why it matters

An agent re-reads the same prefix on every step, which is exactly what prompt caching discounts.

It needs a stable prefix: any change early in the context invalidates everything after it.
Response caching suits repeated questions with fixed answers, not personalised or time-sensitive ones.
Tool caching also protects your own systems from repeated identical calls.
Every cache needs an expiry matched to how fast the underlying data changes.

In the real world: A team found its cache-read tokens were zero because a request id sat in the system prompt; moving it to the last message made nine-tenths of each prefix a cache read.

# 1. Prompt caching: keep the prefix byte-for-byte identical; mark where it ends.
reply = client.messages.create(
    model=MODEL, max_tokens=16000,
    system=[{"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}}],
    tools=TOOL_DEFS,                  # stable order, no per-request values
    messages=messages,                # changing content comes after the cached prefix
)
print(reply.usage.cache_read_input_tokens)     # zero means something in the prefix changed

# 2. Tool result caching: a short time-to-live, keyed by tool and arguments.
def cached_tool(name: str, args: dict, ttl_seconds: int):
    key = f"tool:{name}:{stable_hash(args)}"
    if (hit := cache.get(key)) is not None:
        return hit
    value = TOOLS[name](**args)
    cache.set(key, value, ex=ttl_seconds)
    return value

TTL = {"get_policy": 3600, "search_catalogue": 60, "get_order": 0}   # 0 = never cache

Caching a response that depended on who asked. A cached answer about 'my order' served to another user is a data leak. Include the user in the key, or do not cache user-specific content.

Practise it: Design an AI Coding Assistant

Model routing and smaller models

What it is

Model routing (or tiering) sends each piece of work to the cheapest model that does it well:

a small model for classification, routing, extraction, summarising and grading;
a medium model for routine agent steps;
a large model for hard reasoning, planning and the final decision on risky actions.

Two ways to route: by step type, fixed in code; or by a cascade, trying the small model first and escalating when a check fails.

Why it matters

Price differences between tiers are large, often ten to fifty times.

Most steps in most tasks are routine: reading a result and choosing the obvious next tool.
A planner-executor design (module 7) is tiering by construction: one large call plans, small ones execute.
Before building a cascade, also measure the simpler option of the stronger model at a lower reasoning effort; one model keeps one cache and is easier to run.
Judge by cost per completed task: a cheaper model that needs more steps or fails more can cost more overall.

In the real world: A support agent uses a small model to route and summarise, a medium one for its loop, and a large one only to decide refunds above ₹200; its bill falls by two-thirds with success unchanged.

Visual example

Tiers

One call with 10,000 input tokens and 300 output tokens.

What each example tier costs for the same call

tier	input price per million	output price per million	cost of the call	use for
small	₹25	₹125	₹0.29	Routing, extraction, grading
medium	₹250	₹1250	₹2.88	Routine agent steps
large	₹1250	₹6250	₹14.38	Planning, hard decisions

TIER_FOR = {
    "route": "small", "extract": "small", "summarise": "small", "grade": "small",
    "execute_step": "medium",
    "plan": "large", "decide_refund": "large",
}

def model_for(step_type: str, attempt: int = 0) -> str:
    tier = TIER_FOR[step_type]
    if attempt > 0 and tier != "large":            # cascade: escalate after a failed check
        tier = {"small": "medium", "medium": "large"}[tier]
    return MODELS[tier]

Routing everything to the smallest model to hit a cost target, without measuring success rate. If success falls from 94 to 85 percent, the extra failures, handled by people, cost more than the tokens saved.

Limiting loops and making fewer calls

What it is

The third lever is fewer model calls:

Limits: a step cap and a per-task budget, so the worst case is bounded.
Parallel tool calls: independent look-ups in one turn instead of three.
Better tools: one task-shaped tool instead of three low-level ones in sequence.
Stop wandering: detect repeated calls; make tool errors say what to do next.
Skip the agent where a rule or a single call will do: route simple requests to a cheaper path.
Batch work that is not urgent: many providers charge about half for asynchronous batch processing.

Why it matters

Each step you remove takes its whole resent history with it.

Removing a step near the end of a task saves the most, because that step would have carried the longest context.
Limits do not reduce the average much; they remove the catastrophic tail, the one run that spends a hundred times the norm.
Parallel calls reduce both cost and waiting.

In the real world: One runaway overnight task cost more than the rest of the day's traffic combined; a ₹50 per-task budget made the worst case ₹50.

class Budget:
    """A hard ceiling per task, enforced by code before every model call."""
    def __init__(self, max_rupees: float, max_steps: int):
        self.max_rupees, self.max_steps = max_rupees, max_steps
        self.spent, self.steps = 0.0, 0

    def check(self) -> None:
        if self.steps >= self.max_steps:
            raise BudgetExceeded("step_limit")
        if self.spent >= self.max_rupees:
            raise BudgetExceeded("cost_limit")

    def record(self, usage) -> None:
        self.steps += 1
        self.spent += cost_of(usage)

Setting the step limit so high it never triggers. Look at the distribution of steps for successful runs and set the limit a little above the longest; anything beyond is almost always a loop.

The levers applied in order

What it is

Applying the levers to the baseline task, one after another:

Trim tool results, so each step adds less.
Cache the fixed prefix.
Tier: the medium model for the two hard steps, a small model for the rest.
Fewer steps: run independent look-ups in the same turn.

The order matters: free wins first, then the ones that trade something.

Why it matters

Trimming and caching change nothing about behaviour, so they carry no quality risk.

Tiering and step reduction change how the agent works, so each needs the evaluation set to confirm success has not dropped.
The savings multiply rather than add: each lever applies to what the previous one left.
After these four, the next step is usually architectural: a cheaper path for simple requests.

In the real world: A team applies the four levers over three weeks, running the evaluation set after each; cost per task falls by more than nine-tenths and the success rate moves by less than a point.

Visual example

Worked example

Baseline: 10 steps, medium model, 1,500 tokens added per step. Each row keeps the levers above it.

One task's cost as each lever is applied

lever	cost per task	cost of 100,000 tasks	saved against baseline
Baseline	₹26.88	₹26.9 lakh	0%
Trim results to 400 tokens per step	₹14.5	₹14.5 lakh	46%
Cache the fixed prefix	₹8.43	₹8.4 lakh	69%
Medium model for 2 hard steps, small for the rest	₹2.19	₹2.2 lakh	92%
7 steps instead of 10, by parallel look-ups	₹1.98	₹2 lakh	93%

Illustrative prices. The first two rows are free; the last two must be checked against the evaluation set.

Starting with the risky lever. Swapping to a smaller model is tempting because it is one line, but it can lower quality; trimming and caching are safer and often save more.

Cost per resolved task

What it is

The number that matters to the business is not cost per task but cost per resolved task:

cost per resolved task = agent cost + (share of tasks it fails × cost of a person handling one);
a failed task is not free: someone picks it up;
so success rate is a cost input, and often the largest.

Why it matters

It stops false economies.

A change that halves token cost and drops success by five points can raise the true cost.
A change that adds a verification step, costing a little more per task, can lower the true cost by catching failures.
It is the figure to compare against the alternative: a person doing the whole task.
It also tells you when to stop optimising: when the agent's own cost is small beside the cost of its failures, work on success rate.

In the real world: A team that had been praised for cutting token cost by 40 percent found its cost per resolved ticket had risen; the cheaper model failed more, and each failure cost a human ten minutes.

Visual example

True cost

A failed task is handled by a person at ₹60.

Three configurations by cost per resolved task

configuration	agent cost per task	success rate	cost per resolved task
Medium model throughout, nothing optimised	₹26.88	94%	₹30.48
All levers applied, quality held	₹1.98	93%	₹6.18
Small model throughout	₹1.45	80%	₹13.45

The cheapest tokens are not the cheapest outcome: the small model's failures cost more than its tokens save.

Reporting only tokens per task to leadership. It rewards cheap failure. Report cost per resolved task alongside the success rate.

Interview questions: agent cost optimisation

What it is

The questions interviewers ask about agent cost optimisation, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Asked to halve an agent's cost, a candidate says 'first show me the per-step token breakdown'; the interviewer has heard 'use a smaller model' from everyone else that day.

Q: Why are agents expensive?

Each step resends the whole history, so input tokens grow with every step and the same text is paid for repeatedly. A ten-step task costs far more than ten single calls.

Q: How would you reduce an agent's cost?

In order: trim tool results and compress the context; cache the stable prefix; route routine steps to smaller models; reduce steps with parallel calls and better tools; cap steps and budget per task; send simple requests down a cheaper path. Check success on the evaluation set after each quality-affecting change.

Q: What is prompt caching and what breaks it?

The provider stores an unchanged prefix and charges a fraction to re-read it. Any change in the prefix, such as a timestamp in the system prompt or a reordered tool list, invalidates it.

Q: What is model routing?

Sending each kind of step to the cheapest model that handles it: small for classification and extraction, medium for routine steps, large for planning and risky decisions; fixed by step type or as a cascade that escalates on failure.

Q: When is response caching unsafe?

When the answer depends on the user, the time or changing data. Cache keys must include whatever the answer depends on, and entries need an expiry.

Q: Why cost per resolved task?

Because failed tasks are picked up by people. A cheaper configuration with a lower success rate can cost more in total. It is the figure to compare with a human doing the work.

Q: How do you stop a runaway bill?

Per-task step and cost budgets enforced in code, repeat detection, per-user and per-tenant limits, alerts on cost per task and hourly spend, and a way to disable an agent or tool quickly.

Q: Bigger model at lower effort, or a smaller model?

Measure both on the same tasks. A strong model at reduced reasoning effort can match a weaker one's cost with better results and keeps a single cache; a cascade adds complexity and splits the cache.

Proposing a smaller model as the first and only lever.

Exercise: cut an agent's cost and prove quality held

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A finance partner asks every agent team for this table quarterly; the teams that have it get budget, and the ones that quote tokens per call are asked to come back.

Your task. Measure your agent's cost per task, apply three levers one at a time, and report cost per resolved task after each.

Steps

Build a ten-case test set with expected end states and run it three times to get a baseline success rate.
Record, per run, input and output tokens per step and compute cost with a price table of your choosing.
Lever 1: trim your largest tool result. Re-run and record.
Lever 2: restructure the context so the prefix is stable, enable prompt caching, and confirm cache-read tokens are above zero.
Lever 3: move classification and summarising steps to a smaller model.
After each lever, recompute cost per task, success rate and cost per resolved task with a ₹60 cost per human-handled failure.
Add a per-task budget and show a deliberately looping task is stopped by it.

You are done when

You have a four-row table: baseline and each lever, with cost, success and cost per resolved task.
You can say which lever saved the most and which, if any, hurt success.
The looping task ends at the budget with a hand-over, not a crash.

Stretch. Try the smallest model for the whole loop and show, with your own numbers, whether cost per resolved task rises or falls.