Serving for online and batch predictions

WHAT THIS TESTS Whether you recognize online and batch scoring as distinct workloads with opposite optimization targets, and can serve both from a shared model.

THE TWO PATHS Online serving answers single requests synchronously and is optimized for latency. It runs as an always-on, low-latency endpoint behind a load balancer, autoscaled to absorb traffic spikes, fed by an online feature store for fresh per-entity features, and held to a tight p99 budget. Throughput per request is small; responsiveness is everything.

Batch serving scores large datasets at once and is optimized for throughput and cost per prediction. It runs as scheduled or triggered jobs on a distributed engine such as Spark or a batch transform service, reads features from the offline store or warehouse, uses large batch sizes to maximize GPU and CPU efficiency, and can run on cheaper or spot instances since latency per record does not matter.

SHARED FOUNDATION Both paths must load the same versioned model artifact and use the same feature definitions, so an entity scored online and in batch gets consistent results. A feature store with offline and online materialization enforces this and prevents skew.

INFRASTRUCTURE CONSIDERATIONS Online: horizontal autoscaling, warm replicas, connection pooling, low-latency feature lookups, and strict timeouts. Batch: parallelism, large memory and throughput, idempotent restartable jobs, cost-optimized scheduling, and output to a data store.

COMMON WRONG ANSWERS Forcing one architecture to serve both, which either wastes money keeping batch-scale resources always on or fails latency under online load; or maintaining two separate models that diverge.

ONE CONCRETE EXAMPLE A recommendation model serves real-time requests from an autoscaled endpoint with single-digit-millisecond feature lookups, while a nightly Spark batch job rescore the full user base for email campaigns using the same model artifact, large batches, and spot instances to minimize cost.