Your company has 40 teams calling LLM providers directly. Nobody knows the total spend until the invoice arrives, three teams have leaked API keys into client-side code, and when a provider had a 90-minute outage last month every AI feature in the product went down simultaneously.
Build an internal LLM gateway that all traffic routes through. It should centralise credentials, control and attribute cost, provide observability, handle failover between providers, and let a team switch models without changing code.
Current volume is 8 million requests per day, with a peak of roughly 400 requests per second. Requests range from 100-token classifications to 100,000-token document analyses, and about 60% are streaming.
Teams will route around the gateway if it makes their life worse, so latency overhead and ergonomics are product requirements, not nice-to-haves.
Build the architecture on a canvas: place the components, configure them, connect them into a data flow, and write a short reason for each one. The AI reviewer grades your design against a rubric written specifically for this problem.
You have centralised every AI feature behind one service. How is that not strictly worse for availability?
A provider fails 30 tokens into a streaming response. What can your gateway actually do?
A team blows through its monthly budget by 10am. What happens, and how did you enforce it without adding latency?
Minimum 6 components · needs a wide desktop screen