Design ingestion for clickstream and batch product metadata

What's really being asked

This question evaluates whether you understand how to integrate streaming and batch data sources into a cohesive MLOps architecture. The interviewer cares about your ability to reason about data velocity, storage cost, and the feature pipeline layer that turns raw inputs into model-ready datasets.

The full answer

First, acknowledge the dual-velocity nature of the sources. The click events are a high-velocity stream, while the product metadata is a slow-moving daily batch. Second, propose landing the raw stream in a data lake and the batch files in a data warehouse or data lake so that lineage and raw history are preserved. Third, explain that a feature pipeline should sit between raw storage and the model, transforming and joining these inputs into reusable feature sets that can be stored in a feature store. Fourth, discuss format trade-offs in terms of access patterns rather than brand names. For batch metadata and large historical aggregations accessed during training, a format organized by columns allows efficient projection and compression. For the event stream, a format organized by rows minimizes write overhead and simplifies handling of structured and unstructured fields during ingestion. Fifth, mention partitioning strategies such as event-time windows for the stream and date folders for the batch file so that backfills and retrains remain bounded operations.

The mistakes people make

Suggesting a single storage system for both the high-velocity stream and the daily batch file indicates a lack of operational scaling experience. Proposing that the training job read directly from the raw stream and batch file on every run without an intermediate feature pipeline or feature store shows you have not maintained production recommendation systems. Ignoring schema evolution is another major red flag because product catalogs and event payloads change over time. Finally, designing a low-latency real-time join when the metadata only updates daily is over-engineering that adds cost and failure modes without business value.

What usually comes next

How would you handle late-arriving events after the daily batch has already been processed? What is your backfill strategy if a bug is discovered in the product metadata file? How do you prevent training-serving skew when the online model sees recent clicks but the feature store refreshes hourly? The interviewer may also ask how you would partition the data to keep retraining costs predictable as the event volume grows over months.

A concrete example

Imagine you receive ten thousand click events per second and a daily product metadata export of several gigabytes. You would ingest the events through a streaming pipeline into a data lake organized by hour, and you would land the metadata batch in a data warehouse or data lake partitioned by date. An hourly feature pipeline computes interaction aggregates from the event lake, joins them with the latest product categories from the batch store, and writes the resulting features into a feature store. The model training job reads historical feature sets from the store using a columnar-friendly layout for efficient scanning, while the serving layer fetches precomputed features for online recommendations.