Deep Learning Deployment

Deployment freezes a model artifact, tokenizer or preprocessing, runtime dependencies, and input/output schema into one versioned inference contract. A serving system validates requests, bounds size and rate, batches compatible work when useful, applies timeouts, records versions, and supports rollback to a known-good release. Monitoring covers latency, errors, resource use, input drift, output behaviour, safety signals, and delayed quality labels; each measure needs an owner and a response threshold. Offline evaluation, shadow traffic, canary rollout, and rollback are successive ways to limit the blast radius of a bad release.

A model that is accurate offline can fail in production from preprocessing skew, changed inputs, resource exhaustion, or a silent version mismatch. The surrounding system is part of model correctness because it determines the actual tensors the model sees and the conditions under which users receive its result. A reproducible artifact and a monitored feedback loop turn deployment from a one-time upload into an operating discipline.

The Machine Learning syllabus's production discipline applies unchanged: a versioned artifact, a validated schema, canary rollout, rollback, monitored drift. Four things are genuinely different. The artifact is gigabytes rather than megabytes, so loading is a meaningful part of startup and storage is a real cost. Inference may need a GPU, which is expensive, scarce, and cannot be autoscaled as casually as CPU. A single request's compute is large enough to matter individually. And throughput depends on *batching*, which introduces a latency-throughput tradeoff that a gradient-boosted tree simply does not have.

These four differences change the shape of the operational problem rather than adding detail to it. A scikit-learn service scales by adding cheap stateless replicas; a GPU service cannot, because a replica costs an order of magnitude more and one may hold the only copy of a 14 GB model that takes 90 seconds to load. So capacity planning becomes a real engineering activity, cold starts become a product concern, and the question 'CPU or GPU?' has to be answered per model with measurements rather than by default. Everything else you learned about production ML still holds — this is the delta, not a replacement.

Batch inference scores a large set of inputs on a schedule and stores the results for later lookup. Real-time inference scores one request when it arrives, inside a user-facing latency budget. The choice is not about the model; it is about whether the input exists before the request does. A daily recommendation refresh can be batch because the users and items are known in advance; a search ranker cannot, because the query is created by the request itself.

Batch is dramatically cheaper and simpler when it is possible, and the possibility is worth checking before building a real-time service. A batch job can use large batch sizes for near-perfect GPU utilisation, tolerate a failure by retrying, run on spot instances, and has no latency budget at all — so a model far too slow to serve online may be entirely practical offline. The cost of batch is staleness: a prediction computed at 03:00 is wrong by the time a user's behaviour changes at 14:00. That is the actual decision — how stale is acceptable — and many products that built a GPU-backed real-time service would have been better served by a nightly job and a key-value store.

A trained nn.Module is a Python object that depends on your source tree, your package versions and your import paths. Serving it directly means shipping and running that whole environment. Exporting converts the model into a self-contained, framework-level graph instead: TorchScript serialises it into a form the PyTorch runtime can execute without your Python classes, and ONNX serialises it into a vendor-neutral graph that other runtimes — ONNX Runtime, TensorRT, mobile and browser engines — can load and optimise.

Two reasons, and the second is the bigger one. Portability: an exported graph decouples the artifact from your repository, so a refactor cannot break a deployed model and the serving image does not need your training code. Optimisation: an export is a *graph*, which a runtime can rewrite — fusing a convolution with its batch-norm and activation into one kernel, folding constants, selecting hardware-specific implementations. Those rewrites need the whole computation visible ahead of time, which eager Python execution cannot provide. That is where a large part of inference speedup comes from, and it is unavailable to a model served as a live Python object.

A GPU running one request at a time is almost idle — the arithmetic is a fraction of what the hardware can do, and most of the step is fixed overhead. Dynamic batching puts a short queue in front of the model: requests arriving within a small window are collected, run as one batch, and the results are split apart again. Throughput rises steeply because the fixed cost is amortised over many requests, and the marginal cost of one more item in a batch is small.

The tradeoff it creates is the core operational decision of GPU serving. Every request now waits for the batching window before its compute even starts, so p50 latency rises, and the request that arrives first waits longest. Throughput and latency move in opposite directions and you have to choose a point: a 5 ms window might double throughput for 5 ms of added latency, while a 50 ms window might quadruple it at a cost no interactive product can absorb. The right window is derived from the latency budget, not tuned for maximum throughput — pick the largest window whose p99 still fits the SLA, and no larger.

Rollback requires that the previous artifact still exists, still loads, and can be made live quickly. For a deep network each of those has a cost the classical case does not. The artifact is gigabytes, so keeping ten versions is terabytes of storage. Loading takes a minute or more, so rollback is not instant unless the previous version is kept warm. And the artifact is more than weights: the tokenizer, preprocessing, and export must roll back together or you have recreated the version-mismatch failure from module 19.

The practical consequence is that a rollback plan for a deep learning model has to be designed, not assumed. The pattern that works is a retention policy with tiers — keep the last few versions immediately loadable, a longer history in cheaper storage, and every version's manifest forever, since a manifest is bytes and is what lets you reconstruct which model produced a given logged prediction. The other half is keeping the previous replica set alive during a rollout rather than replacing it, so a rollback is a traffic-weight change taking seconds instead of a cold load taking minutes. That costs double capacity during the rollout window, and it is worth it.

The Machine Learning syllabus established the signals: input drift, output distribution shift, latency, errors, and delayed quality labels reconciled as cohorts mature. All of them apply. Deep learning adds one that classical ML rarely needs — GPU cost and utilisation as a first-class monitored metric, tracked per request and per model version, with its own threshold and its own owner.

It is first-class because it is large enough to change decisions. A GPU-backed service can cost an order of magnitude more per request than a CPU one, so a 30% utilisation regression is a budget event, and a latency regression is simultaneously a user-experience problem and a cost problem — slower requests hold the GPU longer, so the same traffic needs more replicas. It is also a *leading* indicator in a way the quality metrics are not: a change in input length distribution shows up as rising GPU time per request days before it shows up in any accuracy number, because more tokens cost more compute immediately and only degrade quality once they exceed what the model handles well. Watching cost per request is watching the input distribution with a much shorter lag.

A deployed deep learning model is a cycle, not a delivery. Serve; log every request with its inputs, its outputs, its latency, its GPU cost and its model version; watch the label-free signals continuously; reconcile against true quality as each cohort matures; and when a threshold is crossed, go back — to the data, to the compression settings, to the batching window, or to the architecture. The drifted segment or the regressed slice is already identified, which is a far better starting point than the original project had.

It is the shape of the whole syllabus. The loop reuses everything: the tokenizer contract from module 19, the fine-tuning discipline from 21, the checkpoint hygiene from 22, the memory budget from 23, the triage method from 24, and the per-slice evaluation gate from 25. And it reuses the Machine Learning syllabus's monitoring loop wholesale, with one metric added — GPU cost per request — that turns out to move first. Twenty-six modules exist so that when the dashed line on the closing chart starts rising, you know which of them to open.