Where to place feature transformations: client, serving API, or upstream service?

What's really being asked

Whether you understand the tension between data consistency, system latency, and operational maintainability when moving feature engineering from training into production serving paths. Interviewers want to see you map technical constraints to business outcomes like release velocity and model reliability.

The full answer

Four dimensions in order. First, client-side transforms, noting they create training-serving skew when client libraries drift from training code, force duplication across mobile web and backend stacks, and complicate rolling out fixes. Second, colocated serving API transforms, which eliminate network hops and reduce RPC latency but couple CPU-heavy work to the inference hot path, making autoscaling harder and increasing tail latency under load. Third, a dedicated upstream feature service, which centralizes logic, enables reuse across multiple models, simplifies versioning and auditability, but introduces a network hop and a new failure domain. Fourth, hybrid or context-aware placement, such as putting latency-sensitive lookups in the serving API while pushing heavy aggregations to the upstream service.

The mistakes people make

Treating this as a purely latency question and ignoring consistency. Proposing client-side transforms without mentioning version drift or skew. Claiming a dedicated service is always best without acknowledging the operational burden of an extra tier. Failing to discuss what happens when training code changes and serving code lags.

What usually comes next

How would you enforce that training preprocessing exactly matches serving preprocessing? What is your strategy for versioning feature schemas when multiple models consume the same service? How do you handle A-B testing two transform implementations without duplicating infrastructure? When would you push transforms into the model graph itself, for example using TensorFlow Transform?

A concrete example

Imagine a recommendation model needing a user click sequence embedding and a real-time inventory availability join. A senior candidate places the embedding lookup in a dedicated feature store service because it is shared across three models and changes weekly, while performing the lightweight inventory join inside the serving API to avoid an extra network call for a latency-sensitive checkout flow. They keep no transforms in the mobile client beyond basic input validation.