How would you architect dynamic training resource provisioning?

WHAT THIS TESTS: This question evaluates whether you can design elastic infrastructure rather than static clusters. The interviewer wants to see that you understand the trade-off between training throughput and infrastructure cost, and that you know how to automate the full lifecycle of ML compute from provisioning to teardown. Key signals are awareness of GPU scarcity, spot instance economics, checkpointing for fault tolerance, and event-driven automation.

A GOOD ANSWER COVERS: First, a metadata-driven router or heuristic that inspects dataset size, model parameter count, and target latency to select the right instance family before any hardware is provisioned. Second, on-demand provisioning through managed training services or Kubernetes cluster autoscaler so nodes exist only for the duration of the job rather than running idle. Third, aggressive use of spot or preemptible instances combined with frequent checkpointing to S3 or persistent volumes, which can reduce compute costs by up to 70 percent while surviving interruptions. Fourth, automatic teardown triggered by job-state events such as SUCCESS or FAILURE rather than idle timeouts, ensuring resources are released the moment training ends. Fifth, a feedback loop that compares requested resources against actual GPU and memory utilization to refine future sizing decisions.

COMMON WRONG ANSWERS: Proposing a fixed-size GPU cluster that runs continuously and is scaled manually when a big job arrives. Suggesting serverless functions like AWS Lambda for the actual training workload, which is architecturally invalid for GPU-bound tasks. Ignoring checkpointing when using spot instances, which means a preemption destroys hours of progress. Relying on cron-based or idle-timeout scaling instead of event-driven teardown, which leaves expensive GPUs burning cash between jobs. Failing to separate the sizing decision from the provisioning step, leading to either over-provisioning or out-of-memory crashes.

LIKELY FOLLOW-UPS: How would you handle a tenfold increase in dataset size without rewriting the training code? What happens if a spot instance is reclaimed mid-epoch? How do you coordinate multi-node distributed training when individual nodes are provisioned dynamically? How do you prevent cold-start latency from eating the cost savings on small jobs? How do you enforce budget caps or queue limits when many teams share the same elastic pool?

ONE CONCRETE EXAMPLE: A pipeline receives a fine-tuning request. A lightweight classifier reads the dataset metadata and selects a ml.g5.xlarge for a small job or a ml.p4d.24xlarge for a 10B parameter model on 50TB of data. It submits a SageMaker Training Job with checkpointing to S3 and spot capacity. When the job emits a SUCCEEDED state to EventBridge, a Lambda function deletes the training cluster and moves artifacts to long-term storage. If the spot instance is interrupted, SageMaker automatically resumes from the latest checkpoint on a new instance. The entire GPU fleet exists only for the duration of the job.