Agentic AI · Module 24 of 41 · 10 concepts · about 16 min

Scaling Agent Systems

What changes when one agent becomes thousands running at once: concurrency and asynchronous execution, queues and workers, horizontal scaling, rate limits and quotas, shared state and distributed locking, and where the bottlenecks really are.

At work: The work between a pilot with fifty users and a product with fifty thousand, where the limiting factor is almost never your servers and almost always someone else's rate limit.

Why agents scale differently

What it is

An agent's workload has a different shape from an ordinary web request:

Long-lived: seconds to hours, not milliseconds.
Mostly waiting: nearly all the time is spent waiting for a model or a tool, not computing.
Variable: one task takes three steps, the next takes thirty.
Stateful: each run has state that must survive between steps.
Limited by outside services: the model provider's rate limits, and the capacity of the systems the tools call.

Why it matters

These properties change what 'scaling' means.

Because runs mostly wait, one worker can hold many runs at once if it does not block while waiting.
Because runs are long, the number in flight at once matters more than requests per second.
Because the true limit is external, adding servers does not help once the provider's quota is reached.
Capacity planning starts from tokens per minute, not from CPU.

In the real world: A team doubled its servers to fix slow responses and saw no change; the delay was requests queueing against the model provider's tokens-per-minute limit.

Visual example

Capacity

8 steps per task; the context starts at 2,500 tokens and grows by 600 per step; peak is 4 times the average.

Sizing for 200,000 tasks a day

quantity	value	how
Average tasks per second	2.3	200,000 ÷ 86,400
Peak tasks per second	9.3	average × 4
Seconds per task	23	8 steps × (model call + 400 ms tool)
Runs in flight at peak	213	peak tasks per second × seconds per task
Tokens per task	38,000	all inputs resent each step, plus outputs
Peak tokens per minute	21.1 million	peak tasks per minute × tokens per task
Peak model requests per minute	4,444	peak tasks per minute × 8
Workers needed	16	runs in flight × 1.5 headroom ÷ 20 runs per worker

The row to negotiate with your model provider is tokens per minute. The worker count is small by comparison.

Scaling an agent service like a web API: more instances behind a load balancer. The instances sit idle waiting on the model, and the provider returns 'rate limit exceeded' regardless of how many you run.

Practise it: Design an AI Customer Support Agent

Concurrent agents and asynchronous execution

What it is

Two terms:

Concurrency is handling many runs at once.
Asynchronous execution means that while one run waits for a model or tool reply, the worker gets on with another instead of sitting idle. In Python this is async and await.

A semaphore caps how many operations of one kind run at once: 'at most 50 model calls in flight'.

Why it matters

Since agents spend their time waiting, asynchronous code lets one worker serve dozens of runs.

A blocking worker handles one run at a time, so a thousand concurrent runs need a thousand workers.
An asynchronous worker handles many, so the same load needs a few dozen.
A semaphore keeps you under a provider's limit and keeps one large job from starving the others.
Within a run, the same mechanism executes independent tool calls together.

In the real world: Moving a worker from threads to asynchronous code took its capacity from about forty concurrent runs to several hundred on the same machine.

import asyncio

MODEL_SLOTS = asyncio.Semaphore(50)          # at most 50 model calls in flight per worker

async def call_model(messages, tools):
    async with MODEL_SLOTS:                  # wait here, without blocking other runs
        return await client.messages.create(model=MODEL, max_tokens=16000,
                                            messages=messages, tools=tools)

async def run_agent(run_id: str) -> None:
    state = await store.load(run_id)
    while state.status == "running" and state.step < MAX_STEPS:
        reply = await call_model(state.messages, TOOL_DEFS)
        results = await asyncio.gather(*[execute(c, state) for c in reply.tool_calls])
        state = advance(state, reply, results)
        await store.save(state)              # checkpoint: any worker can continue from here

async def worker() -> None:
    async with asyncio.TaskGroup() as group:              # many runs on one worker
        async for run_id in queue.subscribe():
            group.create_task(run_agent(run_id))

One blocking call inside asynchronous code: a synchronous database driver or time.sleep. It freezes every run on that worker until it returns. Every wait must be awaited.

Practise it: Design an AI Customer Support Agent

Queues and workers

What it is

The standard shape for agent workloads:

requests become jobs on a queue;
a pool of workers takes jobs and runs them;
the queue length tells you whether you have enough workers;
priority queues let interactive chats go ahead of batch jobs;
per-tenant limits stop one customer's burst from delaying everyone (the noisy neighbour problem);
jobs that keep failing go to a dead-letter queue.

Why it matters

A queue separates the rate at which work arrives from the rate at which you can do it.

A burst waits in line instead of failing.
Backpressure: when the queue is long you can slow intake, shed low-priority work or tell users to expect a delay, instead of collapsing.
A worker that dies loses nothing: its job becomes visible again and another worker resumes from the checkpoint.
Workers scale on a signal that matters: queue depth and wait time.

In the real world: A single customer's bulk import once delayed every other customer's chats by ten minutes; per-tenant concurrency limits and a separate batch queue made it invisible the next time.

Visual example

Queue and workers

Arrival rate and working rate are decoupled by the queue.

How a burst is absorbed

Burst300 requests in a minute
QueueInteractive and batch lines
WorkersMany jobs each, within limits
AutoscaleWait rises: more workers
DrainLine empties; workers removed

def enqueue(run_id: str, tenant_id: str, interactive: bool) -> None:
    if tenant_in_flight(tenant_id) >= TENANT_LIMIT[tenant_id]:      # fairness between tenants
        raise TooManyRuns("This account has reached its concurrent-run limit.")
    queue_name = "interactive" if interactive else "batch"
    queue.send(queue_name, {"run_id": run_id, "tenant_id": tenant_id},
               max_attempts=3, dead_letter="failed-runs")           # no endless retries

# Workers read 'interactive' first and take 'batch' jobs only when it is empty.

One shared queue for everything. An overnight job of fifty thousand documents then sits in front of a customer waiting for a chat reply. Separate queues by urgency.

Horizontal scaling and distributed agents

What it is

Two ways to add capacity:

Vertical scaling: a bigger machine.
Horizontal scaling: more machines doing the same job.

Agents scale horizontally when workers are stateless: everything about a run lives in a shared store, so any worker can pick up any run. Distributed agents are then simply runs spread across many workers, possibly in several regions.

Why it matters

Horizontal scaling is what gives both capacity and resilience.

Workers can be added and removed freely, including in the middle of runs.
A failed worker costs one step, not a task.
The requirement is strict: no run state in a worker's memory, and no assumption that step five runs on the same machine as step four.
What does not scale by adding workers: the model provider's quota, and the downstream systems your tools call.

In the real world: A platform runs its workers on short-lived, cheap machines that can be withdrawn at two minutes' notice; because each step is checkpointed, withdrawals are invisible to users.

Keeping a cache of run state in worker memory 'for speed'. It works until a run's next step lands on a different worker, which with horizontal scaling is most of the time.

Practise it: Design an AI Customer Support Agent

Rate limits and API quotas

What it is

Every external service caps how much you may use. For model providers the limits are usually:

requests per minute (RPM);
tokens per minute (TPM), input and output;
sometimes concurrent requests and a daily or monthly quota.

Exceeding one returns an error (HTTP 429, 'too many requests'), usually with a header saying how long to wait. Your tools' own back ends (payment services, third-party APIs) have limits too.

Why it matters

The quota is the real ceiling on your throughput.

Agents consume tokens fast, because each step resends the growing history.
At the limit, more workers make things worse: more requests are rejected and retried.
So you must shape your own traffic: a rate limiter in your model gateway that admits calls at the rate the provider allows, with priority for interactive work.
Retries on a 429 must wait, with backoff and jitter (increasing, slightly randomised delays), or the retries themselves keep you over the limit.

In the real world: A launch-day surge sent an agent service into a retry storm; a shared token bucket in the model gateway and jittered backoff turned a two-hour outage pattern into a ten-second queue.

import asyncio, random

class TokenBucket:
    """Admit work at the provider's rate. Capacity refills continuously."""
    def __init__(self, tokens_per_minute: int):
        self.capacity = self.level = tokens_per_minute
        self.rate = tokens_per_minute / 60
        self.updated = now()

    async def take(self, tokens: int) -> None:
        while True:
            self.level = min(self.capacity, self.level + (now() - self.updated) * self.rate)
            self.updated = now()
            if self.level >= tokens:
                self.level -= tokens
                return
            await asyncio.sleep((tokens - self.level) / self.rate)     # wait for a refill

async def call_with_backoff(request, attempts: int = 5):
    for attempt in range(attempts):
        try:
            await bucket.take(estimate_tokens(request))
            return await send(request)
        except RateLimited as error:
            wait = error.retry_after or min(60, 2 ** attempt)
            await asyncio.sleep(wait + random.uniform(0, 1))            # jitter: do not retry in step
    raise ProviderUnavailable()

A rate limiter in each worker's memory. Ten workers each allowing the full quota send ten times the quota. The limiter's count must be shared, usually in a cache such as Redis.

Shared state and distributed locking

What it is

With many workers, two can touch the same thing at once. A race condition is a bug that depends on which gets there first. Two tools prevent it:

A distributed lock lets only one worker hold a named lock at a time: 'the run for conversation 4411'. It has an expiry (a lease) so a crashed worker does not hold it for ever.
Optimistic concurrency: each saved state carries a version number. A save succeeds only if the version has not changed since it was read; otherwise the worker reloads and retries.

Why it matters

Agents make the classic problem sharper.

A user sends two messages quickly: without a lock, two workers run the same conversation and both issue the refund.
A queue redelivers a job after a timeout while the first worker is still running it.
Two agents update the same shared record and one overwrites the other.
Locks and version checks prevent the overlap; idempotency keys make the overlap harmless if it happens anyway. Use both.

In the real world: Customers who tapped 'send' twice occasionally got two refunds; a per-conversation lock removed the overlap, and idempotency keys on the refund tool would have caught any that slipped through.

async def run_exclusively(conversation_id: str) -> None:
    """Only one worker may run a conversation at a time."""
    lock = f"lock:conversation:{conversation_id}"
    token = uuid4().hex
    # SET if Not eXists, with a 60-second lease so a crashed worker cannot hold it for ever
    if not await redis.set(lock, token, nx=True, ex=60):
        return                                   # another worker has it; the message stays queued
    try:
        await run_agent(conversation_id, on_step=lambda: redis.expire(lock, 60))   # renew the lease
    finally:
        await release_if_owner(lock, token)      # never release a lock you no longer hold

async def save(state) -> None:
    """Optimistic concurrency: fail if someone else saved first."""
    updated = await db.execute(
        "UPDATE runs SET data = $1, version = version + 1 WHERE id = $2 AND version = $3",
        state.json(), state.run_id, state.version)
    if updated == 0:
        raise ConcurrentUpdate(state.run_id)

A lock with no expiry. The worker holding it crashes and the conversation is frozen for ever. Every lock needs a lease, and long work must renew it.

Practise it: Design a Multi-Agent Incident Response System

Production bottlenecks

What it is

In rough order of how often each is the real limit:

Model provider quota: tokens per minute.
Model latency: each step waits seconds, in sequence.
Downstream systems: the orders API allows a hundred requests a second; a thousand agents ask for more.
Context growth: long runs resend ever-larger histories, using quota faster.
State store: a checkpoint write on every step of every run.
Vector search: retrieval under load.
Human approval queues: people do not scale by adding machines.
Cost: the budget, which is a limit like any other.

Why it matters

Scaling effort is wasted unless it is aimed at the actual bottleneck.

Measure where time and quota go before changing anything.
Most of the high-ranking items are relieved by using fewer tokens and fewer steps, not by adding infrastructure: smaller contexts, cheaper models for easy steps, caching, parallel tool calls.
Agents can also overload your own back ends, so tools need their own rate limits and caches.

In the real world: A team's agents brought down their own internal search service; a cache and a concurrency cap in the search tool fixed what looked at first like a model problem.

Visual example

Bottlenecks

Notice how few of the remedies are 'add servers'.

Where agent systems run out, and what relieves each

bottleneck	symptom	relief
Provider tokens per minute	429 errors; queues grow	Smaller contexts; model tiering; higher quota; a second provider
Model latency	Slow tasks	Fewer steps; parallel tools; faster model for easy steps; streaming
Downstream APIs	Tool timeouts under load	Per-tool rate limits; caching; batching
Context growth	Cost and latency rise with task length	Trim results; compaction
State store	Slow checkpoints	Smaller state; store bulk elsewhere; partition by run
Approval queues	Runs wait for hours	Raise thresholds; better approval screens; more approvers
Budget	Spend limit reached	Per-task caps; cheaper tiers; caching

Optimising your own servers when traces show ninety-five percent of each run is spent waiting on the model. Your code is rarely the slow part.

Graceful degradation

What it is

Graceful degradation means doing less, deliberately, when capacity is short, instead of failing completely. Options, mildest first:

Queue non-urgent work and tell users the expected wait.
Fall back to a smaller or alternative model.
Reduce effort: fewer retrieval rounds, no reflection step.
Disable expensive optional features.
Shed load: reject low-priority requests with a clear message.
Hand over to people for the cases that cannot wait.

A circuit breaker automates part of this: after repeated failures of a dependency, calls to it are stopped for a while instead of piling up.

Why it matters

Under overload, a system with no plan fails in the worst way: slowly, for everyone.

Deciding in advance what to drop means the important work continues.
A circuit breaker stops a failing dependency from consuming every worker with timeouts.
Users tolerate 'this will take a few minutes' far better than a spinner that never ends.

In the real world: During a provider incident an agent service switched to a smaller model for answering questions and paused all refund actions until the main model returned; customers saw slower refunds and no wrong ones.

Degrading silently into wrong answers: switching to a weaker model for a task it cannot do, with no change to what the agent is allowed to act on. If quality drops, autonomy should drop with it.

Interview questions: scaling agent systems

What it is

The questions interviewers ask about scaling agent systems, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Asked to scale an agent to a million tasks a day, the candidates who start by multiplying tokens per task by peak tasks per minute reach the real constraint in thirty seconds.

Q: How would you scale to thousands of concurrent agents?

Stateless asynchronous workers pulling from queues, all run state in a shared durable store, a shared rate limiter in a model gateway sized to the provider's tokens-per-minute quota, per-tenant concurrency limits, idempotent tools and per-run locks. Then cut tokens per task, because quota is the ceiling.

Q: What is the main bottleneck?

Usually the model provider's tokens-per-minute limit, then model latency and the downstream systems tools call. Rarely your own compute.

Q: Why asynchronous execution?

Runs spend almost all their time waiting on models and tools. Asynchronous workers hold many waiting runs each, so a few dozen workers serve what would need thousands of blocking ones.

Q: How do you handle provider rate limits?

Shape traffic with a shared token bucket, prioritise interactive work, retry 429s with backoff and jitter honouring retry-after, fall back to another model or provider, and reduce tokens per task.

Q: Two workers pick up the same conversation. What happens?

With a per-conversation lock, the second backs off. With versioned saves, the second's write fails and it reloads. Idempotency keys ensure that even if both ran a write, the effect happens once.

Q: How do you prevent one tenant affecting others?

Per-tenant concurrency and rate limits, separate queues or weighted fair scheduling, per-tenant budgets, and tenant ids on every metric so a noisy neighbour is visible.

Q: How do you size capacity?

From tasks per day to peak tasks per second, times seconds per task for runs in flight, times tokens per task for tokens per minute. Compare with the provider quota and derive worker count with headroom.

Q: What do you do under overload?

Queue and communicate, fall back to cheaper models for safe tasks, cut optional steps, shed low-priority work, and use circuit breakers on failing dependencies, reducing autonomy if quality falls.

Answering with 'Kubernetes autoscaling' and nothing about provider quotas.

Exercise: size and load-test an agent service

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A platform team runs this as a game day each quarter against the real system: burst traffic, a killed worker and a throttled provider, all at once.

Your task. Do the capacity arithmetic for a workload, then build a small asynchronous worker pool with a shared rate limit and see it hold under a burst.

Steps

For a workload of your choice, compute peak tasks per second, runs in flight, tokens per minute and model requests per minute, as in this module's table.
Write a fake asynchronous model call that sleeps one to three seconds and fails with a rate-limit error above a set number of calls per second.
Write an asynchronous worker that runs fake eight-step agents from a queue, checkpointing a counter after each step.
Submit five hundred runs at once with no limiter and record the failure rate.
Add a token bucket and jittered backoff; submit again and record throughput and failures.
Kill the worker half-way and restart it; confirm runs resume from their checkpoints.
Submit the same conversation id twice at once and show a lock prevents both from running.

You are done when

You have a capacity table with every figure derived from stated inputs.
With the limiter, failures fall to near zero and throughput matches the allowed rate.
No run restarts from step one after the worker is killed.
The duplicate submission produces one run, not two.

Stretch. Add a second tenant sending ten times the traffic and implement per-tenant fairness so the first tenant's wait time does not change.