You are building the internal model serving platform for a company with 200 ML models in production, owned by 25 teams. Today each team packages its own Flask container, and the results are predictable: no consistent monitoring, GPU utilisation around 15%, deploys that take a week, and three outages last quarter caused by a model container OOMing on a shared node.
The platform must let a team go from a trained model artifact to a production endpoint without talking to you. It must serve both CPU models (mostly GBDTs, single-digit ms) and GPU models (transformers, 50-500ms), support canary rollouts, and drive GPU utilisation up substantially.
Traffic is spiky: some endpoints see 5,000 QPS at lunchtime and 20 QPS overnight. Others are called twice a day by a batch job.
Build the architecture on a canvas: place the components, configure them, connect them into a data flow, and write a short reason for each one. The AI reviewer grades your design against a rubric written specifically for this problem.
GPU utilisation is 15%. Give me the three changes with the largest impact, in order, and estimate each.
Why is CPU-based autoscaling the wrong signal for an inference service?
A model's p99 doubles at 25% canary traffic but error rate is unchanged. What does your system do?
Minimum 8 components · needs a wide desktop screen