Differences between on-demand, reserved, and spot EC2 instances?

What's really being asked

This question evaluates whether you understand EC2 purchasing models beyond their names. The interviewer wants to see if you can balance cost, availability guarantees, and operational complexity when provisioning infrastructure for machine learning. It also reveals whether you think about workload characteristics like fault tolerance, duration, and criticality before picking a compute tier.

The full answer

First, define the three models in terms of commitment and interruption risk. On-demand provides instant capacity with no commitment and highest per-second cost, making it ideal for short-lived or unpredictable workloads like exploratory data analysis and prototyping. Reserved instances require a one or three year commitment to specific instance families in exchange for a significantly lower hourly rate, which suits always-on or predictable pipelines such as critical production model training that runs on a fixed schedule. Spot instances let you use spare EC2 capacity at a steep discount but with a two-minute interruption warning when AWS reclaims the capacity, so they fit only fault-tolerant distributed training or batch inference jobs that checkpoint progress regularly. Second, map each to concrete ML scenarios: on-demand for Jupyter notebooks and ad-hoc experiments, reserved for recurring retraining or persistent inference endpoints that must stay up, and spot for large-scale offline batch inference or hyperparameter search where individual node failure does not kill the job. Third, mention hybrid strategies, such as running a training cluster with a reserved primary node for stability and spot workers for elasticity, or using on-demand as a fallback during spot droughts.

The mistakes people make

Recommending spot instances for real-time model serving or low-latency inference because the cost is low; this ignores the two-minute interruption window and potential latency spikes. Claiming reserved instances are always cheaper without acknowledging the lock-in risk or the need for steady-state utilization to break even. Treating on-demand as a default for everything and dismissing reserved or spot options entirely. Failing to mention checkpointing or retry logic when proposing spot for training workloads.

What usually comes next

How would you handle a spot interruption during a multi-hour training run? When does a Savings Plan make more sense than a reserved instance? How do you automate the fallback from spot to on-demand in a Kubernetes cluster? What metrics would you track to decide if a reserved instance purchase is justified?

A concrete example

A team runs a nightly four-hour deep learning training job that is not time-critical but must complete before morning dashboards refresh. They launch the job on a cluster of GPU spot instances with checkpointing to S3 every epoch. If spot capacity is interrupted, the training script catches the termination signal, saves the latest checkpoint, and the orchestrator reschedules the job on fresh spot or on-demand capacity. Over a month, this cuts compute spend by roughly half compared to on-demand while still meeting the SLA, whereas the production real-time inference API sits on reserved instances to guarantee uptime.