Data Collection and Logging

ML systems draw on a few kinds of source:

• Product events: what users see and do in the app. Views, clicks, searches, additions to the basket.
• Business records: the company's own databases. Orders, payments, refunds, stock.
• Operational data: what the business's systems and people do. Rider GPS pings, packing times, support tickets.
• Human labels: people judging examples. Reviewers marking content, annotators tagging images.
• External data: bought or public. Weather, holidays, maps, traffic.

Each source has its own quality, delay and owner.

• Product events are plentiful but noisy, and depend on the app logging them correctly.
• Business records are accurate but may change after the fact (an order is refunded a week later).
• External data can disappear if a contract ends.
• A design should name the source of every important feature and label.

An event log for ML must record enough to rebuild each decision later. For every prediction shown to a user, log:

• who and when: user ID, session ID, timestamp;
• what was shown: every item, in its position (the impressions), not only what was clicked;
• the context: device, page, location, the search query;
• which model produced it, and its version;
• a request ID that joins the prediction to everything the user did next.

Training data is built from this log, and anything not logged is lost for ever.

• Without impressions there are no negative examples.
• Without position you cannot correct for the fact that top items get clicked more just for being on top (module 7).
• Without the model version you cannot tell which model caused a change.

When a model makes a live prediction, also log the exact feature values it used, under the same request ID. Later:

• join the logged features to the label when it arrives;
• train the next model on those rows.

This is called log and wait, or feature logging.

It is the most reliable way to make training data match production.

• The model trains on exactly the values it saw live, including any quirks of the live pipeline.
• It removes a whole class of bugs where training recomputes a feature slightly differently (training-serving skew, module 11).
• It also lets you debug a single bad prediction months later.

An event schema says what fields an event has, their types and meanings. A data contract is an agreement between the team producing data and the teams using it:

• the schema, and which fields are required;
• what each value means, with units ('amount in paise, not rupees');
• how fresh the data will be;
• how changes are announced, and what counts as a breaking change.

ML pipelines break silently when upstream data changes.

• An app release renames a field and the feature quietly becomes empty.
• A team changes 'amount' from rupees to paise and every price feature is 100 times too big.
• Contracts, checked automatically, turn these into loud failures at the source.

Data reaches ML systems in two ways:

• Batch (data at rest): collected over a period and processed together, such as yesterday's orders loaded into the warehouse each night.
• Streaming (data in motion): each event processed as it arrives, within seconds, through a message system such as Apache Kafka or Amazon Kinesis.

Most systems need both.

• Training uses batch data: large, complete, easy to process.
• Live features need streaming: 'items added in the last ten minutes' cannot wait for a nightly job.
• Streaming is more complex and costly to run, so use it only where freshness changes predictions (module 10).

Three places data is kept for analysis and training:

• Data warehouse (BigQuery, Snowflake, Redshift): cleaned tables with fixed columns, queried with SQL. Fast, tidy, more expensive per byte.
• Data lake (files in S3 or Google Cloud Storage): anything, in raw form, cheaply. Flexible, but can turn into a swamp nobody understands.
• Lakehouse (Delta Lake, Apache Iceberg, Apache Hudi): table features such as updates, versions and schemas on top of cheap lake files.

Training needs both raw history and clean tables.

• Raw events in the lake let you rebuild features you did not think of yet.
• Clean tables in the warehouse or lakehouse make daily training jobs simple and fast.
• Table formats with time travel (reading the table as it was on a date) make training sets reproducible.

Two orders for preparing data:

• ETL (extract, transform, load): clean and reshape data before storing it in the warehouse.
• ELT (extract, load, transform): store the raw data first, then transform it inside the warehouse with SQL (often with a tool such as dbt).

ELT has become the common default because storage is cheap and warehouses are powerful.

• Keeping raw data means a transformation bug can be fixed and re-run.
• Transformations written as versioned SQL can be reviewed and tested like code.
• ETL still suits sensitive data that must be cleaned (for example, personal details removed) before it is stored.

Privacy rules apply as soon as data is collected, not only when it is used:

• Consent and purpose: data collected for delivery may not automatically be usable for advertising.
• Minimisation: collect only what the feature needs.
• Personal data: names, phone numbers, exact locations need protection: removal, masking or encryption.
• Retention: delete data when it is no longer needed, and when users ask.

Laws such as India's Digital Personal Data Protection Act and Europe's GDPR set these duties.

Fixing privacy later is expensive or impossible.

• A model trained on data that should not have been used may have to be thrown away.
• A user's request for deletion must reach every copy, including training sets.
• Designing it in from the start (separate keys, masked fields, retention jobs) costs little.

New products often have no history. Ways to start anyway:

• Rules and heuristics first, while logging begins.
• Public or bought data to pre-train (product images, maps, weather).
• Human labelling of a small, well-chosen set.
• Transfer from a related product or market: a model trained in one city, adapted to another.
• Exploration: deliberately show varied options and learn from the responses.

Waiting for perfect data stalls projects.

• A rule plus good logging produces training data as a by-product.
• Starting small and well-labelled often beats starting large and noisy.
• The cold-start problem returns for every new user and new product, so the solutions are worth designing in (module 13).

The questions interviewers ask about data collection and logging, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Reading an idea and explaining it are different skills.

• Each answer below is the two or three sentences an interviewer is listening for.
• The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

An idea you have only read fades in a week.

• Building it once, even in fifty lines, is what makes it yours.
• Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.