Bezen

ML System Design · Module 20 of 47 · 12 concepts · about 14 min

Deployment, Release and Rollback

Getting a new model into production safely: what CI/CD means when data and models change as well as code, shadow and canary releases, blue-green switches, automatic rollback, verifying model files, kill switches, and keeping a record of every release.

At work: The step where an offline win meets real traffic. Most serious ML incidents trace back to a release that went to everyone at once with no quick way back.

Why model releases need extra care

What it is

A model release can go wrong in ways a code release cannot:

  • it can be wrong without erroring: every request succeeds, with worse predictions;
  • it can be wrong only for some users: one city, one device, new customers;
  • it can depend on features that are missing or different in production;
  • its harm may show only later: fraud losses weeks after a weaker fraud model goes live.

Why it matters

So releases must expose a new model gradually and watch the right signals.

  • Error rates and latency are necessary but not enough.
  • The checks must include the predictions themselves, and business metrics where possible.

In the real world: A delivery company once pushed an ETA model trained with a units bug to all users; no errors appeared, but every estimate was badly off until customers' complaints revealed it hours later.

Releasing a model through the same pipeline as a code change, judged only by 'no errors after deploy'. A model that doubles every delivery estimate passes that check.

CI/CD for machine learning

What it is

In ordinary software only code changes. In ML, three things change, and each needs testing:

  • code: unit tests, as usual;
  • data: validation of every training set (module 8);
  • model: the promotion gate (module 15), plus serving checks: it loads, it answers, it is fast enough, it agrees with offline predictions on a fixed set of examples.

Continuous delivery for ML (CD4ML) automates all three, so a new model can move from training to production through the same tested path every time.

Why it matters

Automation makes releases routine and repeatable.

  • Manual steps are skipped under time pressure.
  • A fixed path means every model in production passed the same checks.
  • It also makes frequent retraining practical.

In the real world: Kirana Kart's release pipeline loads every candidate into the actual serving container and checks its predictions on 1,000 stored requests against the offline predictions before any traffic reaches it.

Visual example

Release path

Each stage can stop the release automatically.

From a registered model to full traffic

  1. No users affected

    RegisteredPassed the offline gate
  2. Serving checksLoads, fast enough, matches offline
  3. Shadow0% of usersScores live traffic; output unused

  4. Real users, step by step

    Canary1% → 50%Small share, watched closely
  5. A/B testMeasures the business effect
  6. Full rollout100%Old model kept ready

Testing the model only offline. A model that scores well offline but fails to load in the serving image, or returns different predictions there, is caught only by a check run in the serving environment.

Shadow deployment

What it is

In a shadow deployment the new model receives a copy of live requests and makes predictions, but its outputs are not used:

  • the old model still answers every request;
  • the new model's predictions are logged beside the old ones;
  • compare them: how often they differ, by how much, for which users; and check latency and errors under real load.

Why it matters

It tests the new model on real traffic with zero risk to users.

  • It catches serving bugs, missing features, skew and latency problems before anyone is affected.
  • Where labels arrive later, the shadow's predictions can be scored against them, giving a true comparison on live data.

In the real world: Banks commonly run a new fraud model in shadow for weeks, until enough chargebacks arrive to compare it fairly with the live one.

Letting a shadow model take real actions. A shadow fraud model that also writes to the blocklist is not a shadow; make sure every side effect is disabled.

Practise it: Design a Real-Time Fraud Detection System, Design Continuous ML Deployment

Canary releases

What it is

A canary release sends a small share of real traffic to the new model, then raises it in stages:

  • start at 1–5% of traffic;
  • watch errors, latency, prediction distributions and key business metrics, compared with the old model;
  • if healthy for a set time, raise the share (5%, 25%, 50%, 100%);
  • if not, roll back automatically.

Why it matters

A bad model affects only a few users before it is caught.

  • Canary analysis compares canary against the rest statistically, rather than eyeballing graphs.
  • The schedule trades speed of rollout against exposure to risk.

In the real world: Kirana Kart releases ETA models at 1%, 5%, 25%, 50% and 100%, holding each stage for at least an hour at peak, with automatic rollback if the canary's average error is worse.

Visual example

Computed

The ETA service at peak (347 requests a second). If a bad model takes 10 minutes to detect, this is how many requests it touches at each stage.

requests affected = share × requests a second × seconds to detect
stageshare of trafficrequests in 10 minutes
11%2,083
25%10,417
325%52,083
450%104,167
5100%208,333

Going straight to 100% would expose every request in the detection window.

A canary too small or too short to detect anything. At 1% of a low-traffic service for ten minutes there may be too few requests to see even a large problem.

Blue-green deployment

What it is

In a blue-green deployment two complete environments exist:

  • blue serves all traffic with the old model;
  • green is brought up with the new model, tested, and warmed up;
  • traffic is switched from blue to green in one step;
  • blue stays running, so switching back is instant.

Why it matters

It makes the switch and the rollback fast and clean.

  • No mix of versions during the switch.
  • The new environment can be fully tested before it receives traffic.
  • It is often combined with canaries: shift traffic to green gradually rather than all at once.

In the real world: Kirana Kart keeps the previous model's replicas running for 24 hours after a full rollout, so a rollback is a routing change.

Tearing down blue right after the switch. The rollback you will need is to blue, and it should take seconds, not a rebuild.

Automatic rollback

What it is

Define in advance what triggers a rollback, and let the system do it:

  • errors or latency above limits;
  • prediction distribution far from the old model's (average ETA up 30%; fraud flag rate doubled);
  • business or proxy metrics worse beyond a threshold;
  • fallback rate rising (features missing, timeouts).

Why it matters

Humans are slow at 2 a.m.

  • An automatic rollback within minutes limits the damage of a bad release.
  • Writing the triggers forces the team to decide what 'bad' means before release, not during an incident.

In the real world: Kirana Kart's canary controller rolls back if the canary's share of orders flagged as fraud differs from the control's by more than half, a check that has fired twice on models trained on broken data.

Rollback triggers only on errors. The most common bad ML release produces no errors at all; include a check on the predictions themselves.

Practise it: Design Continuous ML Deployment

Verifying model files

What it is

A model file is executable in effect: it decides what the system does. Protect it like code:

  • load models only from the registry, never from ad hoc paths;
  • record and check a checksum (a fingerprint of the file) before loading;
  • sign model files in sensitive systems;
  • prefer safe formats: Python pickle files can run arbitrary code when loaded; formats such as ONNX, safetensors or framework-native files are safer.

Why it matters

A swapped or corrupted model file can quietly change decisions, or run malicious code.

  • Checksums catch corruption and the wrong file.
  • Signatures and safe formats close a real attack route (module 25).

In the real world: Hugging Face introduced the safetensors format and scans uploaded files, because pickled model weights were a practical way to hide code.

import hashlib

def load_verified(entry):
    """entry comes from the model registry: path, sha256 and format."""
    data = storage.read_bytes(entry.path)
    if hashlib.sha256(data).hexdigest() != entry.sha256:
        raise RuntimeError(f"Checksum mismatch for {entry.name} v{entry.version}; refusing to load")
    return load_from_bytes(data, fmt=entry.format)   # never pickle.loads on untrusted bytes

Downloading pretrained models from the internet and loading them with pickle. Malicious model files that run code on load have been found on public model hubs.

Kill switches and decoupled releases

What it is

Separate releasing a model from deploying code:

  • the serving code reads which model version to use from configuration or the registry;
  • a feature flag controls who gets the new model;
  • a kill switch turns an ML feature off entirely, falling back to rules or defaults.

Why it matters

Changing a model should not require a code deploy, and turning it off should take seconds.

  • Product, risk or operations staff can disable a misbehaving model without an engineer.
  • Code and model can be rolled back independently.

In the real world: Kirana Kart's risk team has a kill switch for the fraud model; flipping it moves decisions to a rules-only policy, which they have used during a major data outage.

Baking the model file into the application image. Every model change becomes a full release, and rollback means redeploying the whole service.

Releasing a model that needs new features

What it is

When a new model needs a feature the old one did not, release in order:

  • first deploy the feature pipeline and serve the feature (unused);
  • check it is fresh and matches its training values (replay test);
  • then release the model that uses it;
  • remove old features only after no live model reads them.

Why it matters

Releasing both at once couples two risks.

  • If the feature is missing at serving, the new model receives defaults and silently underperforms.
  • Ordered releases make each step checkable and reversible.

In the real world: Kirana Kart's registry lists the features each model version reads, and the platform refuses to retire a feature that any deployable version still needs.

Removing a feature from the pipeline because the newest model does not use it, while the rollback model still does. The first rollback then fails.

Audit trail and approvals

What it is

Keep a record of every release:

  • which model version went live, when, at what traffic share;
  • who approved it, and the evidence (gate results, experiment results);
  • every rollback and its reason.

For high-stakes models (credit, health, hiring), add a human approval step with named reviewers.

Why it matters

Records answer the questions asked after something goes wrong.

  • 'What changed at 3 p.m. on Tuesday?' should take a minute to answer.
  • Regulators in finance and other fields expect documented model governance.

In the real world: Banks follow formal model-risk guidance (such as the US Federal Reserve's SR 11-7) that requires independent validation and documented approval before a model is used for decisions.

Visual example

Strategies

Most teams chain them: shadow, then canary, then A/B.

Release strategies compared
strategyusers affectedwhat it tells youspeed of rollback
ShadowNoneServing health; agreement with old model; later, true accuracyNothing to roll back
CanaryA small, growing shareProblems under real use, earlySeconds (route away)
Blue-greenAll, at the switchClean cut-overSeconds (switch back)
A/B testA planned shareEffect on business metricsSeconds (end the test)

Approvals that are rubber stamps. An approval step only helps if the approver sees the evidence that matters: slices, guardrails and what changed.

Practise it: Design Continuous ML Deployment

Interview questions: deployment and release

What it is

The questions interviewers ask about deployment and release, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

  • Each answer below is the two or three sentences an interviewer is listening for.
  • The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: MLOps interviews nearly always ask for a release plan; product ML interviews ask 'how would you roll it out?' near the end.

Q: How would you release a new model safely?

Pass the offline gate, run serving checks in the real image, shadow it on live traffic, canary it on a small growing share with automatic rollback, then A/B test the business effect before full rollout, keeping the old model ready.

Q: What is shadow deployment?

The new model scores a copy of live traffic but its outputs are not used. It reveals serving bugs, skew and latency, and later true accuracy once labels arrive, with no risk to users.

Q: Canary or blue-green?

Canary shifts a small, rising share of traffic and watches it; blue-green keeps two full environments and switches all at once with instant switch-back. They are often combined.

Q: What should trigger an automatic rollback?

Errors and latency, but also prediction distributions far from the old model's, worse proxy or business metrics, and a rising fallback rate.

Q: How do you release a model that needs a new feature?

Deploy and verify the feature first, unused, then release the model; retire old features only when no deployable model needs them.

Q: Why avoid pickle for model files?

Loading a pickle can execute arbitrary code. Load only from the registry, verify checksums or signatures, and prefer safe formats.

Releasing to 100% straight after offline evaluation.

Practise it: Design a Model Serving Platform, Design a Real-Time Fraud Detection System, Design Continuous ML Deployment

Exercise: build a canary with automatic rollback

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

  • Building it once, even in fifty lines, is what makes it yours.
  • Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A release controller like this is a staple project for ML platform teams.

Your task. Route a share of traffic to a new model, compare it with the old one live, and roll back automatically when it misbehaves.

Steps

  1. Run two versions of your model service (from module 18) side by side.
  2. Write a small router that sends a configurable share of requests to the new version, consistently per user.
  3. Log predictions from both, and run the new model in shadow on 100% of traffic as well.
  4. Write a controller that every minute compares error rate, p99 latency and the average prediction between canary and control, and raises the share in stages if healthy.
  5. Release a deliberately broken model (for example one that multiplies predictions by 1.5) and confirm the controller rolls it back.
  6. Record every stage change and rollback in an audit log.

You are done when

  • A healthy model reaches 100% through all stages without intervention.
  • The broken model is rolled back automatically, with no errors needed to trigger it.
  • The audit log tells the full story of both releases.

Stretch. Add a checksum check so a model file altered on disk is refused at load time.

Practise it: Design Continuous ML Deployment