Retraining and Feedback Loops

A model is a snapshot of the world when its training data was collected. It decays as the world moves:

• customer tastes and habits change;
• the business changes: new cities, products, prices, app screens;
• the environment changes: seasons, events, competitors;
• adversaries adapt to the model itself.

Measure the decay: train once, then score each following week, and plot error against weeks since training.

The decay curve tells you how often to retrain.

• A slow curve: monthly retraining is fine.
• A steep one: daily retraining, or fresher features, or both.
• It also tells you how bad things get if retraining fails for a week.

Combine a schedule with triggers:

• a schedule sized from the decay curve;
• a performance trigger: retrain when measured accuracy (or a trusted proxy) falls past a limit;
• a drift trigger: retrain when important inputs drift persistently;
• an event trigger: retrain after a known change (a new city launch, a price overhaul, a new app screen).

Schedules are reliable; triggers react to the unexpected.

• A schedule alone misses a sudden change between runs.
• Triggers alone depend on monitoring being right, and can fire on noise.
• Before retraining on a trigger, check the drift is real and not a data bug (module 21).

Choosing the training window is a trade-off:

• longer windows give more data and cover rare events (last year's festival);
• shorter windows reflect the current world better;
• weighting recent examples more than old ones gives some of both;
• keeping seasonal data (the same weeks last year) helps models that must handle annual events.

There is no universal answer; test it.

• Backtest models trained on 3, 6, 12 and 24 months and compare error on recent weeks.
• Fast-changing problems (trends, fraud) prefer short windows; stable ones benefit from more data.

The champion is the model in production; a challenger is a candidate trying to replace it:

• every retrain produces a challenger;
• it must beat the champion on the same recent data (the gate, module 15);
• then prove itself on live traffic (shadow, canary, A/B; modules 17 and 20);
• only then does it become the champion; the old champion stays ready for rollback.

It makes retraining safe by default.

• A retrain that is worse, because of bad data or bad luck, never reaches users.
• The same process covers routine retrains and new model designs.

A feedback loop forms when a model's predictions influence the data it will be trained on next:

• a recommender shows item A more, so A gets more clicks, so the next model shows A even more;
• a fraud model blocks certain orders, so their outcome is never seen, so the next model never learns whether blocking was right;
• a predictive policing or credit model concentrates attention on a group, generating more records about that group.

Loops can amplify small, accidental differences into large ones.

• Popular items get more popular whether or not they are better.
• The data becomes narrower, so the model knows less and less about anything it stopped showing.
• The model's offline scores can look better while the system gets worse, because it is being tested on data it shaped.

The standard defences:

• exploration: show a small, random share of other options (module 14), so the model keeps learning about them;
• holdouts: a small random group whose outcomes are observed without the model's influence (a few orders not blocked but reviewed; a few users shown non-personalised results);
• propensity logging: record how likely each item was to be shown, and re-weight training data by its inverse;
• position and popularity debiasing in training (module 7);
• monitor diversity: how many distinct items get meaningful exposure.

Each defence restores some information the loop would destroy.

• Exploration and holdouts generate unbiased data.
• Propensity weighting corrects for the bias in the rest.
• The cost is a small, measured loss of short-term performance.

In fraud, spam, abuse and security, people actively study and evade the model:

• a fraud pattern that is blocked is replaced by a new one within days or weeks;
• spammers test messages until they pass;
• a model's own rules leak through its decisions.

Adversarial problems decay fastest.

• Retraining must be frequent, and features that are hard to fake (device, network, behavioural patterns) matter more than easily changed ones.
• Layers help: rules for known patterns, a model for general risk, anomaly detection for the new.
• Fast feedback channels (reviewers, user reports) become essential, since final labels arrive late.

Online learning updates a model continuously from a stream of new examples, every few minutes or hours, rather than in periodic full retrains:

• the model adjusts to change almost immediately;
• it suits fast-moving signals: trending items, ad click rates, news.

Many teams use a middle path: frequent incremental updates from the last full model, with a periodic full retrain.

It wins where freshness is everything, but adds risk.

• A bad batch of data affects the model within minutes.
• Testing every update is harder.
• Models can drift into strange states over many small updates.
• Safeguards: validation on every update, automatic rollback, and regular full retrains to reset.

A mature system runs the full loop automatically, with checks at every step:

• monitoring detects decay or a trigger fires;
• the training pipeline builds a challenger on validated data;
• the gate compares it with the champion;
• release automation shadows, canaries and promotes it;
• monitoring watches the new champion.

People review what the automation did, approve big changes, and handle what it cannot.

Automation is what lets a small team run many models.

• Every step that needs a human becomes a step that is skipped under pressure.
• The safeguards are what make the automation trustworthy; without them, it automates mistakes.

The questions interviewers ask about retraining and feedback loops, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.