Agentic AI · Module 27 of 41 · 10 concepts · about 16 min

Latency Optimisation

Where an agent's time goes and how to get it back: parallel tools, asynchronous execution, caching, streaming, choosing faster models for easy steps, speculative work, and cutting reasoning that is not needed.

At work: The difference between an agent people use and one they abandon: a correct answer that arrives after the user has given up is a failed task.

Where the time goes

What it is

Latency is how long something takes. For an agent task it is the sum of:

model calls, each made of a time to first token (the wait before any output appears) plus generation time (output tokens ÷ tokens per second);
tool calls: your systems' own response times;
overhead: queue wait, loading and saving state, guardrail checks.

The steps of a loop run one after another, so their times add up.

Why it matters

Two facts shape every optimisation.

Model time is driven mostly by output length and number of calls, much less by input length.
Sequential steps multiply: six steps of three seconds is eighteen seconds, however fast each part is.
So the big wins are fewer sequential steps, shorter outputs, and faster models for the steps that do not need a slow one.
Measure two numbers: time to the first visible response, and time to completion. Users feel the first far more.

In the real world: A team profiled a 20-second task and found 14 seconds in six sequential model calls, 5 in tools and under half a second in their own code; the plan that followed touched no application logic.

Visual example

Breakdown

A model call producing 150 output tokens, then a tool taking 500 ms.

What one step costs in time

	small model	medium model
Time to first token	300 ms	600 ms
Generation	1000 ms	1875 ms
Tool call	500 ms	500 ms
One step	1.8 s	3 s
6 sequential steps	10.8 s	17.9 s

Optimising your own code. In a typical trace, model and tool waits are over 95 percent of the time. Shaving milliseconds from the loop changes nothing a user can feel.

Parallel tools and asynchronous execution

What it is

The largest single saving is to stop doing independent things one at a time:

Parallel tool calls: the model requests several tools in one turn; your code runs them together. Three look-ups cost one model call and the slowest tool's time, not three of each.
Parallel sub-tasks: independent strands of work run as separate agents at once (module 16).
Asynchronous execution: awaiting several operations together instead of in sequence.
Non-blocking side work: logging, memory writes and analytics happen after the reply is sent, not before.

Why it matters

Each removed sequential step removes a whole model call from the wait.

Parallel tools need a prompt that encourages it ('request independent look-ups together') and results returned in one message.
Only independent work can be parallel; a refund cannot run beside the check that justifies it.
Moving memory extraction and trace export off the critical path costs nothing and often saves a second.

In the real world: A support agent that fetched order, policy and customer history in one turn instead of three cut its median reply time from nine seconds to four.

import asyncio

async def run_tool_calls(calls, ctx) -> list[dict]:
    """Independent reads run together. The wait is the slowest one, not the sum."""
    if all(call.name in READ_ONLY for call in calls):
        return await asyncio.gather(*[execute(call, ctx) for call in calls])
    return [await execute(call, ctx) for call in calls]       # writes stay in order

async def reply_then_tidy(state, answer: str) -> None:
    await send_to_user(answer)                                 # the user has their answer
    asyncio.create_task(extract_memories(state))               # not on the critical path
    asyncio.create_task(export_trace(state))

Returning parallel results in separate messages, or in a prompt that never mentions parallel use. The model falls back to one tool per turn and the saving disappears.

Streaming

What it is

Streaming sends output to the user as it is produced, instead of waiting for the whole reply. For agents there are three things to stream:

the final answer, token by token;
progress updates: 'Looking up your order…', 'Checking the refund policy…';
intermediate results: a partial table, the plan, sources as they are found.

Why it matters

Streaming does not make the task faster. It makes the wait shorter and the system feel alive.

A user who sees the first words within a second will wait ten more; one facing a blank screen leaves after five.
Progress messages for tool steps turn a silent pause into a visible process, and build trust in what the agent is doing.
Streamed text cannot be unsent, so output checks must run on what is streamed, or a holding message is shown until the answer is verified.

In the real world: Adding three status messages to a twelve-second task changed nothing about its duration and cut the share of users who abandoned it by half.

async def stream_run(task: str, send) -> None:
    """Send progress as the agent works, then stream the answer."""
    async for event in agent.stream(task):
        if event.type == "tool_start":
            await send({"type": "status", "text": STATUS_TEXT.get(event.tool, "Working…")})
        elif event.type == "text_delta":
            await send({"type": "token", "text": event.text})
        elif event.type == "needs_approval":
            await send({"type": "status", "text": "Waiting for a colleague to approve this."})

STATUS_TEXT = {"get_order": "Looking up your order…", "get_policy": "Checking our policy…",
               "refund_item": "Issuing your refund…"}

Streaming an answer that a later check then rejects. The user has already read it. For risky content, validate before streaming or stream only status until the answer passes.

Model selection for speed

What it is

Models differ in speed as well as price. For latency:

use a small, fast model for routing, extraction, grading and simple tool-selection steps;
keep the large model for the steps that need it;
reduce reasoning effort on easy steps; deep thinking on a trivial step is pure delay;
ask for short outputs: output length is the main driver of generation time;
some providers offer faster serving of the same model at a higher price.

Why it matters

A small model can be several times faster to first token and to finish.

Most steps in a task are easy, so most of the time can be bought back.
Shortening the output is free: 'reply in two sentences' or a compact structured output halves generation time.
As with cost, judge on the whole task: a fast model that needs two extra steps is not faster.

In the real world: A router moved to a small model answers in 400 milliseconds instead of 1.8 seconds; because every request passes through it, that one change shortened every task.

Asking for long step-by-step reasoning in the visible output of every step. The user waits for text nobody reads. Keep intermediate outputs terse and let the final answer be the only long one.

Caching for speed

What it is

The caches of module 26 also save time:

Prompt caching: a cached prefix is processed faster, shortening time to first token on long contexts.
Tool result caching: a policy or catalogue look-up served from memory takes a millisecond.
Response caching: an identical or equivalent question is answered instantly.
Pre-computation: fetch what is nearly always needed (the user's profile, recent orders) when the conversation opens, before it is asked for.

Why it matters

The fastest call is the one you do not make.

Many agents fetch the same few things at the start of every task; loading them up front removes a whole step.
Cache expiry must match how quickly the data changes: an hour for a policy, a minute for stock, never for an order's status mid-conversation.
A wrong cached answer is fast and wrong, so cache only what is safe to be slightly stale.

In the real world: Pre-loading the customer's last three orders removed the first tool step from four in five support conversations, taking about three seconds off each.

async def open_conversation(user) -> dict:
    """Fetch what almost every support task needs, before the first model call."""
    profile, recent_orders = await asyncio.gather(
        get_profile(user.id), get_recent_orders(user.id, limit=3))
    return {"preloaded": render(profile, recent_orders)}       # goes into the first context

# The agent often needs no get_order call at all: the order is already in front of it.

Pre-loading everything 'in case'. Ten kilobytes of rarely used data in every context slows and costs every turn. Pre-load only what most tasks use.

Speculative workflows

What it is

Speculative execution starts work before you know it is needed, and discards it if it is not:

Speculative tool calls: begin the likely look-ups while the model is still deciding.
Racing: send the same request to two models or providers and take the first good answer (hedging).
Optimistic paths: run the fast path and a guardrail check at the same time; show the result only if the check passes.
Predicted next step: while a person reads an answer, pre-compute the most likely follow-up.

Why it matters

It trades money for time.

You pay for work that is sometimes thrown away, in exchange for not waiting when it is needed.
It suits steps that are highly predictable and cheap, and requests where speed is worth paying for.
Running a guardrail in parallel with generation, instead of before it, removes the check's delay from every clean request.
Only read-only work may be speculative. An action cannot be un-done because the guess was wrong.

In the real world: A voice assistant runs its safety check alongside the reply and hedges slow model calls after two seconds; its slowest one-in-twenty responses fell from six seconds to three.

async def answer_with_parallel_guardrail(message: str):
    """Start the answer and the input check together; release only if the check passes."""
    check = asyncio.create_task(input_guardrail(message))
    answer = asyncio.create_task(generate(message))

    verdict = await check
    if not verdict.ok:
        answer.cancel()                                # stop paying for a reply we will not send
        return refusal(verdict)
    return await answer

async def hedged(request, timeout: float = 2.0):
    """If the first provider is slow, race a second. Take whichever finishes first."""
    first = asyncio.create_task(provider_a(request))
    done, _ = await asyncio.wait({first}, timeout=timeout)
    if done:
        return first.result()
    second = asyncio.create_task(provider_b(request))
    done, pending = await asyncio.wait({first, second}, return_when=asyncio.FIRST_COMPLETED)
    for task in pending:
        task.cancel()
    return done.pop().result()

Speculating on a write. 'Start the refund while we check eligibility' is not an optimisation, it is a bug with a customer attached. Speculate on reads only.

Limiting unnecessary reasoning

What it is

Much of an agent's time is spent thinking it did not need to do:

Skip the loop for simple requests: route them to a single call or a rule.
Skip planning for short tasks.
Skip reflection unless a test shows it pays.
Lower reasoning effort on routine steps.
Stop early: once the answer is known, do not make a confirming extra call.
Replace model steps with code wherever a rule decides: comparisons, look-ups, formatting.

Why it matters

Every pattern from module 8 buys accuracy with time.

On an easy request that purchase is wasted.
A router that sends 'what are your opening hours?' to a direct answer saves the whole agent loop for most traffic.
Code is thousands of times faster than a model call and never wrong about arithmetic.
The discipline is to add reasoning where measurement shows it helps, not everywhere by default.

In the real world: A fast path for the commonest twenty question types answers in under a second; the full agent, with its planning and checks, is reserved for the requests that need it.

Visual example

Fast path

Most requests never enter the loop.

Routing by difficulty

Classifyfew hundred msSmall fast model
one of four routes
Trivial~10s of msRule or cached answer
Simple~2 sOne call with retrieval
Complexseveral sFull agent loop, streamed
Long-runningminutesBackground job; notify later

A single heavyweight path for every request. The simple four-fifths of traffic then waits as long as the hard fifth.

The levers applied, and setting targets

What it is

Applying the levers to a six-step task, in order:

run the three independent look-ups in parallel;
use a small model for the two routine steps;
serve the policy from a cache;
stream progress and the answer.

And set targets by the kind of interaction: a chat needs a first response within about a second; a background job can take minutes if it reports progress.

Why it matters

The first three shorten the task; the fourth shortens the perceived wait.

Targets should be set on the slow tail (p95), because the average hides the users who wait longest.
Some tasks cannot be made fast. Then change the interaction: acknowledge at once, work in the background, notify when done.
Every lever here except streaming should be re-checked against the evaluation set, since each changes how the agent works.

In the real world: A research assistant that could not get below ninety seconds stopped pretending to be a chat: it now confirms the request, shows live progress and sends a notification, and satisfaction rose.

Visual example

Worked example

Baseline: 6 sequential steps on the medium model, each with a 500 ms tool call.

One task's duration as each lever is applied

lever	time to finish	time to first visible output
Baseline	17.9 s	17.9 s
Three look-ups in one turn	11.9 s	11.9 s
Small model for two routine steps	9.6 s	9.6 s
Policy served from cache	9.1 s	9.1 s
Stream status and answer	9.1 s	0.6 s

Streaming changes only the last column, and that is the column users feel.

Chasing a low average while one request in twenty takes a minute. Those are the users who complain and leave. Optimise and alert on p95.

Interview questions: latency optimisation

What it is

The questions interviewers ask about latency optimisation, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: A candidate asked to speed up a slow agent asks first for a trace with timings; the answer 'the three look-ups are sequential' follows in seconds.

Q: Where does an agent's latency come from?

Sequential model calls, each with a time to first token plus generation time driven by output length, plus tool calls and overhead. Steps add up because they run one after another.

Q: How do you reduce it?

Fewer sequential steps through parallel tool calls and better tools; faster models and lower reasoning effort for easy steps; shorter outputs; caching and pre-loading; a fast path for simple requests; and streaming to cut perceived wait.

Q: Does streaming make an agent faster?

No. Total time is unchanged. It reduces the time to first visible output and shows progress, which is what users experience. Streamed text must be checked before or as it is sent.

Q: When can tool calls run in parallel?

When they are independent reads. Writes, and any call depending on another's result, stay sequential. Results go back in one message.

Q: What is speculative execution in an agent?

Starting likely work before it is confirmed needed: pre-fetching, racing providers, or running a guardrail alongside generation. It trades cost for time and is safe only for read-only work.

Q: Which latency metrics do you track?

Time to first response and time to completion, at p50 and p95, plus per-step and per-tool times. The tail matters more than the average.

Q: A task takes ninety seconds and cannot be shortened. What do you do?

Change the interaction: acknowledge immediately, run it in the background, stream progress, and notify on completion. Do not hold a chat open in silence.

Q: How do cost and latency optimisation relate?

Trimming context, caching, tiering and fewer steps help both. Speculation and hedging improve latency at extra cost. Batch processing lowers cost at the expense of latency.

Proposing faster servers for a workload that spends its time waiting on a model.

Exercise: halve an agent's response time

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A product team's weekly review shows one chart: p95 time to first response. It fell from eleven seconds to under two over a quarter, using exactly these levers.

Your task. Measure where your agent's time goes, apply four levers, and report time to first output and time to completion before and after.

Steps

Add timing to every model and tool call (the traces from module 25 already have it). Give each fake tool a 500 ms delay.
Run a task that needs three independent look-ups and record the per-step timeline.
Enable parallel tool execution and prompt the agent to request independent look-ups together. Re-measure.
Add a one-hour cache for the policy tool and pre-load the customer's recent orders at the start.
Move routing to a smaller model.
Stream status messages for each tool and stream the final answer. Record time to first output.
Run twenty times and report median and worst-case times for the first and last versions.

You are done when

You have a before-and-after table with both metrics.
Time to completion has fallen by about half, and time to first output is under two seconds.
The task's end state is still correct in all twenty runs.
You can show, from timings, which lever saved the most.

Stretch. Run the input guardrail in parallel with generation and measure the saving on clean requests.