Describe the difference between online and batch inference.

What's really being asked

This question probes whether you understand that inference is not a single workload but a spectrum of serving patterns. Interviewers want to see if you can reason about latency requirements, resource allocation, cost models, and infrastructure topology rather than just naming two concepts.

The full answer

First, define online inference as synchronous, request-response prediction serving where a client waits for the result, and batch inference as asynchronous, large-scale processing of accumulated inputs without a client waiting. Second, contrast infrastructure: online needs always-on, autoscaling endpoints such as Kubernetes deployments or serverless containers, often with model warm pools, load balancers, and caching layers to keep P99 latency between ten milliseconds and a few seconds; batch needs job schedulers like Airflow, Kubeflow Pipelines, or managed batch services that distribute work across spot or reserved instances, optimizing for throughput over latency with typical runtimes from minutes to hours. Third, discuss resource patterns: online allocates GPU or CPU per concurrent request and must handle traffic spikes, while batch maximizes cluster utilization by processing partitions in parallel and can tolerate preemption. Fourth, mention cost and SLA tradeoffs: online carries higher per-query cost and demands strict availability SLAs, whereas batch leverages cheaper compute and looser timing guarantees.

The mistakes people make

A red flag is claiming that the only difference is speed or that you can simply run the same model container in both modes without changes. Another mistake is ignoring cold-start latency for online serving or suggesting infinite autoscaling without mentioning cost. Candidates also err by omitting data pipeline differences, such as how batch jobs often require ETL preprocessing while online systems need feature stores with low-latency lookups.

What usually comes next

Expect the interviewer to ask how you would handle a model that must serve both real-time and batch, how you choose instance types for GPU-heavy online traffic, or how you would design a fallback when online latency exceeds an SLA. They may also probe monitoring differences, asking what metrics matter for an online endpoint versus a batch job.

A concrete example

Imagine a fraud detection model. Online inference scores each credit card transaction as it happens, requiring an API with P99 latency under fifty milliseconds running on CPU-optimized instances with autoscaling from ten to one hundred pods. Batch inference re-scores the past twenty-four hours of transactions for regulatory reporting, using a nightly Spark or Argo workflow on spot instances that processes ten million rows in thirty minutes with no client waiting for the output.