Your team serves a 13B-parameter chat model behind an API. Traffic has grown to a peak of 600 requests per second, and GPUs are the majority of the infrastructure bill.
A typical request sends 300 input tokens and asks for 150 output tokens back. Product wants time-to-first-token (TTFT) under 300ms at p95, and each A100-80GB GPU sustains roughly 1,500 output tokens/sec for this model under continuous batching.
Traffic is spiky — a 5x spike during a product launch is expected — and GPUs take minutes to provision, not seconds. Finance wants GPU utilisation up; product wants latency down. Design the serving system.
Build the architecture on a canvas: place the components, configure them, connect them into a data flow, and write a short reason for each one. The AI reviewer grades your design against a rubric written specifically for this problem.
Walk through what happens to a request from arrival to first token, under load. Where can it wait, and for how long?
You need to absorb a 5x spike, but GPUs take minutes to provision. What capacity is standing by before the spike starts, and what do you do with requests once that capacity is full?
A 2,000-token summarisation request and a 50-token chat reply arrive seconds apart. Trace how your scheduler treats them differently, if at all.
Minimum 5 components · needs a wide desktop screen