Describe the difference between online and batch inference.

WHAT THIS TESTS: This question evaluates whether you understand the fundamental serving paradigms in production machine learning and can match infrastructure design to business requirements. Interviewers want to see that you distinguish between synchronous, user-facing prediction paths and asynchronous, data-heavy computation pipelines, and that you can discuss trade-offs in latency, cost, availability, and scaling behavior without defaulting to a one-size-fits-all architecture.

A GOOD ANSWER COVERS: First, define online inference as a real-time, request-response pattern where a client sends a payload and expects a prediction within milliseconds to seconds. These systems typically run as always-on containerized endpoints or serverless functions, require autoscaling to handle traffic spikes, and optimize for p99 latency and high availability. Second, define batch inference as an asynchronous process that applies a trained model to a large accumulated dataset, often on a schedule or triggered by an event. It uses elastic compute clusters that can scale out for the job and to zero afterward, optimizing for throughput, total job cost, and completion time rather than per-request latency. Third, discuss use cases explicitly: online suits fraud detection, recommendation feeds, chatbots, and autonomous systems; batch suits nightly risk scoring, generating embeddings for a content catalog, monthly billing forecasts, and large-scale offline model evaluation. Fourth, contrast infrastructure needs: online needs load balancers, health checks, A-B test routing, and low-latency feature stores; batch needs workflow orchestration, checkpointing, data partitioning, and spot or reserved instance management.

COMMON WRONG ANSWERS: A common red flag is claiming that batch is just online inference run slowly or with queued requests. Another is ignoring cost by proposing always-on GPU clusters for batch workloads that run once per day. Candidates also err by omitting data dependencies: batch jobs usually read from data lakes or warehouses and write results back to tables, whereas online systems rely on low-latency feature retrieval from caches or dedicated feature stores. Failing to mention scaling differences, specifically that online scales with request rate while batch scales with data volume, signals shallow experience.

LIKELY FOLLOW-UPS: Interviewers often push deeper by asking how you would handle cold-start latency in online serving or how to partition data for massive batch jobs. They may ask about hybrid architectures, such as precomputing batch results for a cache while falling back to online inference for fresh entities. You might also be asked to compare serverless inference versus persistent endpoints, or to discuss how feature store consistency requirements differ between the two modes.

ONE CONCRETE EXAMPLE: Imagine a retail company predicting next-week demand. An online endpoint would serve a mobile app showing immediate inventory availability, requiring sub-100-millisecond responses and 99.99 percent uptime. A nightly batch job would instead score millions of SKU-store combinations from a data warehouse, write predictions back to a table, and shut down its compute cluster after ninety minutes. The online system prioritizes latency and availability; the batch system prioritizes total cost and throughput.