How do you guarantee identical feature engineering for training and real-time inference?

WHAT THIS TESTS: This question tests whether you understand that training-serving skew is fundamentally an engineering problem, not a modeling problem. The interviewer wants to see if you recognize that divergent code paths between batch training and online inference are the root cause, and whether you can design infrastructure that guarantees identical feature transformations across both environments. They are looking for systems thinking about data consistency, schema contracts, and deployment safety.

A GOOD ANSWER COVERS: A strong answer should hit four things in order. First, establish a single source of truth for feature definitions, typically through a shared transformation library that both training and serving code import and execute. Second, introduce a versioned feature store where offline batch features and online point lookups are served from the same logical store with consistent serialization formats. Third, implement training-time feature logging so you can replay exactly what the model saw during training against what the serving system produces, enabling automated skew detection. Fourth, enforce strict schema contracts and data validation at pipeline boundaries so that upstream changes do not silently break feature computation in one path but not the other.

COMMON WRONG ANSWERS: The biggest red flag is proposing to keep separate training and serving implementations and relying on manual code reviews or unit tests to keep them in sync. Another weak pattern is focusing only on data distribution drift while ignoring code-level skew caused by different languages or runtimes. Candidates also stumble by suggesting heavy-weight ETL for online inference, which introduces latency that makes real-time serving impossible. Finally, failing to mention schema validation or feature logging reveals a lack of operational rigor.

LIKELY FOLLOW-UPS: The interviewer may push on latency constraints, asking how you guarantee sub-50 millisecond feature lookups while maintaining consistency with batch pipelines. They might ask how you handle schema evolution when adding a new feature without breaking existing serving binaries. Another common thread is cost: how do you validate skew across billions of examples without replaying the full training set every hour. They may also probe fault tolerance, such as how the serving system behaves when the feature store is unavailable.

ONE CONCRETE EXAMPLE: Imagine a recommendation model that computes a user's seven-day click-through rate. During training, a Spark job aggregates clicks from a data warehouse. During serving, a Flink job or online service must compute the same seven-day window from a real-time event stream. To prevent skew, you define the aggregation logic once in a shared Python library that compiles to both Spark UDFs and a Python serving microservice. The feature store holds precomputed batch values for backfill and historical training, while the online path computes from a Kafka topic using the exact same function version. You log the training features to BigQuery, then nightly replay the serving logic against the same user events and assert that 99.99 percent of values match within a floating point tolerance. If a deploy changes the window logic in one path but not the other, the replay job alerts before the model goes live.

Source: developers.google.com