Capstone Project

Your task. A one-page problem statement for TripTime, in the framework of module 1.

Steps

1. State the business goal, the users, and how the prediction is used (shown at booking as an estimate and a promise).
2. Define the target precisely: minutes from pickup to drop-off; decide how to treat zero-length and multi-hour trips.
3. Choose offline metrics (MAE, share of trips longer than the promise, pinball loss) and online metrics you would use in a real launch.
4. Estimate requests a second for a city with 500,000 trips a day and a peak factor of 3, and the training-data size for a year.
5. Write a latency budget for the 50 ms API.
6. Write a rule-based baseline.

You are done when

• A reader can say exactly what the model outputs and how success is measured.
• Your estimates trace back to stated assumptions.
• Your baseline could ship without ML.

Your task. A re-runnable pipeline from raw files to a clean trips table, partitioned by day.

Steps

1. Download at least 18 months of yellow-taxi trip records, including early 2020.
2. Write load, clean and write steps as separate, idempotent functions taking a date (module 8).
3. Add validation: schema, row counts per day, durations and distances in sane ranges, valid zone IDs, no duplicates.
4. Record how many rows each check removes per month, and investigate one month where many are removed.
5. Store the output in a versioned format (Parquet with a version folder, or Delta Lake).

You are done when

• Re-running any day gives identical output.
• A deliberately corrupted file is caught by a named check.
• You can say which data version any later model was trained on.

Your task. Features for each trip computed only from information available at booking time, served from a feature store.

Steps

1. Compute on-demand features: distance between zone centres, hour, weekday, holiday flags.
2. Compute batch features: each pickup zone's median duration to each borough over the previous 28 days.
3. Compute recent-demand features: trips started in the pickup zone in the previous 15 and 60 minutes, using only earlier trips (module 10).
4. Register them in Feast with an offline and an online store, and build the training set with point-in-time joins (module 11).
5. Write a replay test comparing online feature values with the offline ones for 1,000 trips.

You are done when

• No feature uses information from after the trip's pickup time.
• The replay test shows matching values for stable features.
• You can show a leaky join and how much it inflates offline scores.

Your task. A chosen model with a percentile promise and a one-page evaluation report (module 16).

Steps

1. Evaluate the rule baseline.
2. Train linear regression and LightGBM on a time split (train on twelve months, validate on the next, test on the one after).
3. Train LightGBM quantile models at 0.5 and 0.85 for the estimate and the promise.
4. Report MAE, share of trips over the promise and pinball loss overall and by borough, hour and trip length.
5. Run error analysis on the 200 worst predictions and add one feature that fixes a group of them.
6. Add bootstrap intervals for the difference between your model and the baseline.

You are done when

• Your model beats the baseline with an interval that excludes zero.
• The promise is exceeded on about 15% of trips across most slices.
• Error analysis led to at least one measured improvement.

Your task. A pipeline that trains, evaluates, compares with the champion and registers a winner.

Steps

1. Move configuration into a file and steps into functions; run them from Airflow, Dagster or Prefect (or a single script with clear stages).
2. Log runs to MLflow with parameters, per-slice metrics, data version and commit.
3. Write a gate: the challenger must beat the champion on the latest month with no borough worse by more than 3%, and predict in under 5 ms.
4. Run it on successive monthly snapshots and record which challengers are promoted.

You are done when

• A deliberately worse challenger (for example trained on one week of data) is rejected with a clear reason.
• The champion can be rebuilt exactly from its registry entry.

Your task. A FastAPI service returning estimate and promise in under 50 ms at p99, plus a nightly batch job for zone-to-zone estimates.

Steps

1. Load the production model from the registry with a checksum check (module 20).
2. Fetch online features from the feature store in one call; compute on-demand ones from the request.
3. Return the estimate, the promise and the model version; log features and outputs with a request ID.
4. Add a timeout falling back to the batch zone-to-zone table, then to the rule baseline.
5. Load-test with Locust; record p50, p99 and requests a second; profile and fix the slowest step.

You are done when

• p99 under 50 ms at your target load.
• Stopping the feature store produces fallback answers, not errors.
• Every response can be traced to its logged features and model version.

Your task. A release controller that shadows, canaries and rolls back a model on its own.

Steps

1. Run the challenger in shadow on replayed traffic and compare its predictions with the champion's.
2. Route 5%, then 25%, 50% and 100% of requests to it, consistently per rider ID.
3. Roll back automatically if errors, p99 latency or the average predicted duration differ from the champion by more than set limits.
4. Release a deliberately broken model (predictions × 1.4) and confirm rollback.
5. Keep an audit log of every stage and rollback.

You are done when

• A healthy model reaches 100% without intervention.
• The broken model is rolled back with no errors needed to trigger it.

Your task. Monitoring that detects the 2020 shift in travel patterns, and a retraining policy that recovers.

Steps

1. Replay months of trips through the service in order, with the model trained on 2019.
2. Track system, data, prediction and accuracy layers (module 21): feature PSI, prediction distribution, MAE and share over promise per month and per borough.
3. Write alert rules with persistence and see which fire, and when, in March and April 2020.
4. Compare three policies: never retrain, retrain monthly on the last 12 months, retrain on a drift trigger with a short window.
5. Decompose any change in overall error into mix and within-segment effects (module 23).

You are done when

• Your alerts fire within days of the shift, not weeks.
• You can show which retraining policy recovered fastest, and its cost.
• Your decomposition separates traffic-mix change from model decline.

Your task. A design document a reviewer could approve, and a talk you could give in an interview.

Steps

1. Write the document to the eight steps of module 1, with your real numbers, choices and the alternatives you rejected.
2. Include diagrams of the architecture, the training pipeline and the release path.
3. Add a section on failures you hit and how you found them.
4. Add a model card (module 24) with per-borough performance and limitations.
5. Present it aloud in twenty minutes to someone who asks questions; note the questions you struggled with and improve those parts.

You are done when

• Every design choice in the document has a stated reason and a rejected alternative.
• You can answer 'what would you do differently with ten times the traffic?' and 'what breaks first?'.