Your company has 30 LLM-powered features in production: summarisation, classification, extraction, chat, code generation. Every team evaluates differently. Most "evaluate" by having an engineer read 10 outputs and decide it looks fine.
Three incidents in the last quarter shipped quality regressions to production: a prompt change that broke JSON formatting for 8% of requests, a model version upgrade that made summaries 40% longer and blew a downstream context limit, and a retrieval change that silently reduced answer groundedness.
You are building the internal evaluation platform. It must let a team define an eval suite, run it in CI against every prompt and model change, compare versions, and catch regressions before they ship.
The hard part is that most of these tasks have no single correct answer.
Build the architecture on a canvas: place the components, configure them, connect them into a data flow, and write a short reason for each one. The AI reviewer grades your design against a rubric written specifically for this problem.
Which of the three incidents would your platform have caught, and with which tier of evaluator?
Why is 'rate this summary from 1 to 10' a weak eval, and what do you do instead?
Your judge says version B is 3% better on 100 examples. Should the team ship it?
Minimum 8 components · needs a wide desktop screen