ML System Design · Module 18 of 47 · 12 concepts · about 14 min

Serving Patterns: Batch, Online, Streaming and Edge

The ways a prediction can reach a product: computed in advance in batch, on request online, triggered by events in a stream, or on the user's device. How each works, what it costs, how to combine them, and what sits on the path of a live prediction.

At work: One of the first architecture decisions for any ML feature. It decides the infrastructure, the cost and how fresh the predictions can be.

Four ways to serve predictions

What it is

Predictions reach the product in one of four ways:

Batch: compute predictions for everyone on a schedule and store them; the product looks them up.
Online: compute when the product asks, within its request, and return the answer.
Streaming: compute when an event happens, without anyone asking, and push the result where it is needed.
Edge: run the model on the user's phone, browser or a device.

Why it matters

Each suits different needs for freshness, latency and cost.

Batch is cheapest and simplest but only as fresh as its last run.
Online uses the latest context but must be fast and always available.
Streaming reacts to events in seconds without blocking any request.
Edge avoids the network entirely, for speed, offline use or privacy.

In the real world: Kirana Kart uses all four: batch for demand forecasts and 'buy again' lists, online for ranking and ETA, streaming for fraud signals on account activity, and edge for a small on-phone model that predicts which search the user is typing.

Visual example

Patterns

Pick per prediction, not per company.

The four serving patterns compared

pattern	computed	freshness	latency to product	cost	Kirana Kart example
Batch	On a schedule	Hours	A lookup (ms)	Lowest	Demand forecasts; buy-again lists
Online	On request	Now	Model time (ms)	Higher	Ranking; delivery estimate
Streaming	On an event	Seconds	Already waiting	Medium to high	Account-risk score after each login
Edge	On the device	Now	No network	Free per call; hard to update	Search-as-you-type suggestions

Choosing online for everything by default. Many predictions could be computed overnight with no loss, at a fraction of the cost and risk.

Batch prediction

What it is

Batch prediction runs the model over every entity (user, product, store) on a schedule and writes the results to a store the product can read:

a nightly job scores all users or items;
results go to a key-value store or a database table;
the product looks up the stored prediction in milliseconds.

Why it matters

Its advantages are large.

Serving is a simple lookup, so it is fast and very reliable.
Heavy models are fine: there is all night to run them.
Predictions can be checked before anyone sees them.
Its limits: predictions are as old as the last run, and they cannot use anything about the current request or session.

In the real world: Kirana Kart computes each customer's 'buy again' list nightly; storing it for every monthly user takes about three gigabytes.

Visual example

Computed

Storing a nightly 50-item list for each of 5,000,000 monthly users.

12 bytes per item (product ID and score)

users	items each	bytes per item	total
5,000,000	50	12	3 GB

Small enough to hold in memory in a single key-value store.

Scoring everyone, including users who will not visit. Restrict batch scoring to recently active users, and fall back to a default for the rest.

Online prediction

What it is

Online prediction computes a prediction during the product's request:

the product calls a prediction service with the request's details;
the service fetches features, runs the model and returns the result;
all within a latency budget, often tens of milliseconds.

Why it matters

It is needed when the prediction depends on the moment.

The current basket, the address just typed, the store's live queue.
Each request can be scored differently.
The price: the service must be fast at p99, scale with traffic, and have a fallback, because the product waits for it.

In the real world: Kirana Kart's delivery-time estimate is computed online on every checkout, because the address, basket and store queue all change.

Making the product's main request wait for a slow, optional prediction. If recommendations are not essential to showing the page, load them separately so a slow model never delays the whole screen.

Streaming and asynchronous prediction

What it is

Streaming (or asynchronous) prediction runs when an event arrives, not when someone asks:

an event (a login, a payment, a new listing) lands on the stream;
a consumer computes the prediction;
the result is stored or acted on: a risk score updated, a listing flagged for review, a notification sent.

Why it matters

It gives freshness without blocking a user's request.

The prediction is ready, or the action already taken, by the time anyone needs it.
Heavier models are possible, since no screen is waiting.
Suits moderation, alerts, enrichment and scores that later requests will read.

In the real world: Kirana Kart recomputes an account's risk score after every login and address change, in the stream; the payment-time fraud check then reads it instantly.

Using streaming where the result must exist before the user continues. A payment must not go through while its fraud score is still in a queue; that decision needs online scoring.

Practise it: Design a Real-Time Anomaly Detection System, Design a Real-Time Fraud Detection System, Design an ML Feature Store

Hybrid serving

What it is

Most real systems combine patterns, doing the heavy work in advance and the context-dependent work live:

precompute candidates, embeddings and slow features in batch;
compute live only what depends on the request: the final ranking, the last-minute features;
fall back to the precomputed result if the live step fails.

Why it matters

It gets most of the freshness at a fraction of the cost.

The expensive part (scoring the catalogue, building embeddings) happens once a night.
The cheap part (re-ranking 500 items with session features) happens per request.

In the real world: Kirana Kart precomputes each customer's embedding and candidate pool overnight; online, it adds basket-based candidates, filters by live stock and ranks with session features.

Visual example

Hybrid

If the live steps fail, the precomputed list is served.

Heavy work at night, light work per request

Night
Nightly batchEmbeddings, candidates, slow features
StoreOnline store and ANN index
Per request
Request arrivesPrecomputed candidates + live context
Filter and rankStock check; session features
RespondTop 20; log what was shown

Precomputing results that depend on fast-changing facts. A precomputed list that includes products now out of stock must be filtered live, or customers see items they cannot buy.

Edge and on-device prediction

What it is

Edge serving runs the model on the user's device or near it:

phones and browsers: keyboards, camera effects, search suggestions;
devices: cameras, sensors, cars, factory machines.

Models are made small (quantised, pruned, distilled; module 19) and run with tools such as TensorFlow Lite, Core ML, ONNX Runtime or ExecuTorch.

Why it matters

Edge serving gives things a server cannot.

No network delay, so very fast responses.
Works offline.
Data can stay on the device, which helps privacy.
The costs: small models only, varied hardware, and updates depend on users installing them.

In the real world: Phone keyboards predict the next word on the device; Google has described training these models with federated learning so typing data never leaves the phone.

Forgetting that edge models cannot be rolled back instantly. A bad model shipped in an app release stays on some phones for months; ship models separately from app code where possible, with a kill switch.

Practise it: Design a Predictive Maintenance System

The serving stack

What it is

A prediction service is usually built from:

a model server that loads models and runs them efficiently: NVIDIA Triton, TensorFlow Serving, TorchServe, BentoML, Ray Serve, or a simple web service for small models;
on Kubernetes, a layer such as KServe or Seldon for scaling, routing and rollouts;
feature fetching from the online store;
pre- and post-processing, the decision layer and logging.

Why it matters

Model servers solve problems every team would otherwise rebuild.

Batching many requests together for GPUs.
Serving several models and versions side by side.
Health checks, metrics and safe model reloads.
For a small tree model, a plain FastAPI service may be all that is needed.

In the real world: Kirana Kart serves its tree models from small FastAPI services and its neural recommendation models from Triton on GPUs, all deployed through KServe.

from fastapi import FastAPI
import lightgbm as lgb

app = FastAPI()
model = lgb.Booster(model_file=registry.download("eta", stage="production"))

@app.post("/eta")
def eta(req: EtaRequest) -> EtaResponse:
    feats = features.fetch(store_id=req.store_id, customer_id=req.customer_id)   # one batched read
    x = build_row(req, feats)                                                 # same code as training
    minutes = float(model.predict([x])[0])
    log.prediction(req.request_id, model_version=MODEL_VERSION, features=x, output=minutes)
    return EtaResponse(minutes=round(minutes), model_version=MODEL_VERSION)

Adopting a heavy serving platform for one small model. The operations work can outweigh the model; start simple and adopt a platform when several models need it.

Packaging models

What it is

A trained model must be saved in a form the serving system can load safely:

framework formats: LightGBM and XGBoost text or JSON files, PyTorch's TorchScript or exported programs, TensorFlow SavedModel;
ONNX: a common format many runtimes can execute fast;
containers: the model plus its code and libraries in an image, so it runs the same everywhere.

Why it matters

Packaging decides how portable, fast and safe serving is.

ONNX and compiled formats often run faster than the training framework.
Containers end 'it worked on my machine'.
The serving environment's library versions must match those used to save the model.

In the real world: Kirana Kart exports its neural models to ONNX for serving, which cut their CPU latency by about a third compared with running them in PyTorch.

Loading pickled models from anywhere but your own registry. Python pickles can run arbitrary code when loaded; prefer safer formats and verify a model's checksum before loading (module 20).

Timeouts, fallbacks and defaults

What it is

Every online prediction needs a plan for when it is slow or fails:

a timeout shorter than the product can wait;
a fallback prediction: a cached result, a simpler model, a rule, or popular items;
defaults for missing features;
circuit breakers: stop calling a failing service for a while instead of piling on retries.

Why it matters

ML services fail in ordinary ways: deploys, overload, slow feature stores.

With fallbacks, those failures make predictions slightly worse instead of breaking the product.
The fallback rate is itself a key metric: a rising one means trouble.

In the real world: If Kirana Kart's ETA model does not answer in 30 ms, checkout shows the distance-based estimate; customers see a number either way.

Retrying a slow model call inside a tight budget. Two retries of a 40 ms call cannot fit in 50 ms; return the fallback instead.

Caching predictions

What it is

If the same inputs recur, store the prediction and reuse it:

key by everything that determines the output (entity, relevant context, model version);
expire entries when the inputs are likely to have changed;
use it for expensive predictions with repeated inputs: product embeddings, popular search queries, per-store forecasts.

Why it matters

A cache hit costs almost nothing.

Popular search queries repeat constantly; caching their candidate lists cuts load sharply.
The risk is stale or wrong results if the key leaves something out.

In the real world: Kirana Kart caches search candidates for the 10,000 most common queries per store for five minutes; about 40% of searches are served from the cache.

Leaving the model version out of the cache key. After a release, the cache keeps serving the old model's predictions until entries expire.

Practise it: Design a URL Shortener

Interview questions: serving patterns

What it is

The questions interviewers ask about serving patterns, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: Interviewers usually ask 'would you compute this in real time or in advance?' early on. A reasoned hybrid answer is the strongest.

Q: Batch or online prediction?

Batch when predictions depend only on slowly changing data and can be precomputed for known entities: cheaper, simpler, checkable. Online when they depend on the current request or session. Often a hybrid of both.

Q: What is streaming prediction used for?

Reacting to events within seconds without blocking a request: moderation, alerts, risk scores that later requests read.

Q: Why serve on the edge?

No network latency, offline use and privacy. The costs are small models, varied hardware and slow, hard-to-reverse updates.

Q: What happens when the model times out?

Return a fallback within budget: a cached prediction, a simpler model, a rule or popular items. Track the fallback rate; use circuit breakers rather than retries in tight budgets.

Q: What does a model server give you?

Efficient execution and request batching, multiple models and versions, health checks, metrics and safe reloads, for example Triton, TorchServe or TensorFlow Serving, often under KServe.

Q: How would you cache predictions?

Key by every input that affects the output, including the model version, expire when inputs change, and use it for expensive, repeated predictions such as popular queries.

Proposing online serving with no fallback.

Exercise: serve a model two ways

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: Building one service with a real fallback is the quickest way to understand why interviewers keep asking about them.

Your task. Serve the same model as a batch job and as an online service, then add a fallback and a cache.

Steps

Take a trained model from an earlier exercise.
Write a batch job that scores all entities and writes results to a key-value store (Redis or a SQLite table).
Write a FastAPI service that computes the same prediction on request, fetching features and logging each prediction with the model version.
Add a 30 ms timeout around the model call with a fallback to the batch result, then to a rule.
Add a cache keyed by inputs and model version.
Load-test both with a simple tool (Locust or hey) and record p50 and p99 latency.

You are done when

Batch lookup and online prediction agree for the same entity and inputs.
Killing the model inside the service returns fallback answers, not errors.
Changing the model version invalidates the cache.

Stretch. Export the model to ONNX, serve it with ONNX Runtime, and compare latency.