Online Inference: Predictions on Demand

WHY IT EXISTS To provide immediate, real-time answers from a machine learning model. Many applications, especially user-facing ones, cannot wait for a nightly job to run. Online inference was created to serve predictions on demand, enabling interactive experiences that depend on a model's output.

THE MENTAL MODEL Think of online inference as a vending machine for predictions. You insert one request (a user's photo, a credit card transaction) and get one prediction back, right now. This is also called real-time inference. It stands in direct contrast to batch inference, which is like a weekly grocery run—processing a huge list of items all at once, on a schedule, when immediate results aren't needed.

HOW IT WORKS A trained model is packaged and deployed behind a web server as an API endpoint, typically using REST or gRPC. When a client application sends a request with input data, the server receives it, preprocesses the data into a format the model understands (a tensor), runs the model to get a prediction, and sends the result back in the API response. This entire round-trip is expected to complete in milliseconds. The underlying infrastructure involves servers (CPU or GPU), load balancers to distribute traffic, and auto-scaling mechanisms to handle fluctuating demand.

WHEN TO USE IT Use online inference for any synchronous, low-latency application where a person or system is actively waiting for a result. Three common places this appears: first, in fraud detection for financial transactions; second, in recommendation engines that suggest the next product or video; and third, in content moderation systems that must block harmful posts instantly.

WHEN NOT TO USE IT Avoid online inference when latency is not a concern and you can process data in bulk. For tasks like generating a daily analytics report, pre-calculating features for millions of users, or transcribing a large archive of audio files, batch inference is far more efficient and cost-effective. Using an online system for a batch task is a waste of resources.

ONE CANONICAL EXAMPLE A credit card fraud detection system. When you swipe your card, the transaction details (amount, location, time) are sent to a fraud model's API endpoint. The model must return a score or a simple 'approve'/'deny' prediction in under 100-200 milliseconds. If it takes longer, the customer experience at the point of sale is degraded. This low-latency, single-event processing is the essence of online inference.