Design: Real-Time Fraud Detection

Clarify:

• goal: cut fraud losses while declining or delaying as few honest customers as possible;
• scale: about 630,000 payments and refund requests a day; peaks around 22 a second, far higher during sales;
• latency: a decision within 100 ms (p99) at payment time, before the order is confirmed;
• fraud types: stolen cards, fake accounts made for sign-up offers, refund abuse ('item never arrived'), account takeover;
• constraints: a review team of 15 people; decisions must be explainable to reviewers; customers can appeal.

Each fraud type has different signals and labels, so name them.

• The latency limit forces online scoring with precomputed and streaming features.
• Review capacity turns the threshold into an operations question (below).

Turn the goal into an ML task with a decision layer:

• target: probability the payment is fraudulent (confirmed by chargeback, reviewer decision or a later ban);
• decision policy: allow, send to review, or block, by score bands and amount (module 3);
• offline metrics: PR-AUC; precision and recall at the operating thresholds; cost of mistakes (module 16);
• online metrics: fraud loss as a share of payment value; false decline rate; reviewer workload; customer appeals upheld.

Fraud is extremely imbalanced (module 7), so accuracy and ROC-AUC mislead.

• Thresholds come from the cost of each mistake and the review team's capacity, and can change daily without retraining.
• False declines are a real cost: an honest customer declined may never return.

Labels are rare, late and shaped by the system:

• chargebacks arrive over weeks to 90 days (module 6);
• reviewer decisions arrive within a day but only for reviewed cases;
• later evidence: accounts banned for abuse, linked to earlier orders;
• selective labels: blocked payments never produce chargebacks, so their truth is unknown;
• holdout: a small random share of medium-risk payments is reviewed instead of blocked, to keep learning the truth (module 22).

Training only on matured labels means the model learns from patterns that may be months old.

• Fast reviewer labels fill the gap for recent patterns.
• Without a holdout, the model can never learn whether its blocks were right.

The strongest fraud features describe behaviour and links, not single facts:

• velocity (streaming, sliding windows): payments, cards and addresses used by this account, device and card in the last 5 minutes, hour and day (module 10);
• account: age, verified phone, past orders and refunds, past disputes;
• device and network: device fingerprint, emulator signals, IP reputation, distance between IP location and delivery address;
• payment: amount relative to the account's usual, new card on an old account, card country;
• graph: how many accounts share this device, card, phone or address; links to accounts banned before.

Fraudsters can change names and emails easily; devices, payment instruments and networks of linked accounts are harder to fake.

• Velocity captures bursts: many attempts in minutes.
• Graph features catch rings of fake accounts that look innocent one by one.

Three layers, each catching what the others miss:

• rules: known patterns and hard limits, maintained by the risk team ('more than 3 cards on one account in an hour');
• supervised model: gradient-boosted trees on the features above, scoring general risk (module 12);
• anomaly detection: an unsupervised detector for unusual clusters of new accounts or orders, catching new schemes before labels exist;
• decision service: combines the three with amount and customer history into allow, review or block.

Rules are fast to change and explainable; the model generalises; anomaly detection covers the unknown.

• Keeping the decision in its own service lets the risk team change thresholds and rules daily.

Two thresholds turn the score into actions:

• block above the high threshold, automatically;
• review between the two: a person decides, seeing the top reasons;
• allow below the low threshold.

The low threshold must produce no more reviews than the team can handle; otherwise the queue grows and customers wait.

Review capacity is a hard constraint.

• Module 16's cheapest threshold (0.3) would flood a 15-person team.
• Raising the review threshold to fit capacity is a business decision, with the cost made visible: some fraud between the thresholds now gets through.
• Better models raise precision in the band, so the same team catches more.

Scoring sits on the payment path, so it must be fast and safe when it fails:

• features fetched in one batched call; the model is a few hundred trees, under 5 ms;
• fail open or closed by amount: if scoring fails, allow small payments, hold large ones for review (module 4);
• streaming velocity features with freshness checks and defaults (module 10);
• the account risk score is precomputed in the stream after logins and profile changes, so payment-time scoring reads it instantly (module 18).

A fraud service outage must not stop all orders, nor open the doors entirely.

• Splitting by amount limits both kinds of damage.
• Precomputation keeps the payment-time path short.

Fraud models are hard to evaluate quickly:

• offline: on matured labels only; time split; PR-AUC, recall at fixed precision, cost at thresholds; slices by fraud type, city, payment method, new accounts;
• shadow: run the challenger beside the champion for weeks until chargebacks mature, then compare on live data (module 20);
• online: holdout-based comparison of loss and false-decline rates; reviewer overturn rate as a fast signal.

Label delay means a real verdict on a new model takes weeks.

• Shadow mode gives that verdict without risk.
• Fast proxies (overturns, appeals, disputes) catch obvious regressions in days.

Fraud models face people actively looking for gaps:

• monitor: score distribution, flag and block rates by segment, reviewer overturns, appeals, chargeback rate by cohort age (module 21);
• anomaly alerts: sudden spikes in new accounts from one device type, one area or one offer;
• retrain weekly plus on triggers; layered defences so one evasion does not open everything (module 25);
• do not reveal exact reasons to suspected fraudsters; reviewers see them, customers get general explanations.

Fraud patterns shift within days after a defence is deployed.

• Rules give same-day responses to new attacks; the model catches up at the next retrain.
• Cohort-based monitoring avoids false alarms from immature labels.

The questions interviewers ask about the fraud design, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.