A 30-person ML team ships models from Jupyter notebooks. Training happens on whoever's laptop or ad-hoc GPU box is free. When a model in production behaves strangely, nobody can reproduce how it was trained: the notebook has been edited since, the data has changed, and the person who trained it has left.
Last month a model was retrained and quietly got worse. It took three weeks to discover that an upstream table had changed schema and a feature was silently null for 40% of rows.
Build the training pipeline platform. A data scientist should be able to go from experiment to a reproducible, scheduled, monitored training job without becoming a platform engineer.
Build the architecture on a canvas: place the components, configure them, connect them into a data flow, and write a short reason for each one. The AI reviewer grades your design against a rubric written specifically for this problem.
List everything you must pin to reproduce a training run exactly. What breaks if you miss each one?
An upstream schema change makes a feature 40% null. Where in your pipeline does that stop, and how?
A retrained model is worse than the one in production. What happens automatically?
Minimum 6 components · needs a wide desktop screen