Design a cost-aware ML training platform for heterogeneous hardware

WHAT THIS TESTS: This question probes your ability to design systems that bridge high-level ML user needs with low-level infrastructure economics. The interviewer cares about three things: whether you can hide hardware complexity behind meaningful abstractions, whether you understand that cost optimization is a function of both time-to-train and the price of the underlying compute, and whether you can reason about scheduling as an optimization problem with real-world constraints like preemption, topology, and migration overhead.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, a device-agnostic workload specification that lets users declare resources in logical units like accelerator-hours, memory, and interconnect bandwidth rather than naming specific chip types. Second, a performance predictor or profiling layer that estimates training step time for a given model architecture and batch size on each target hardware type, because a TPU pod and an A100 cluster have wildly different throughputs for the same PyTorch script. Third, a normalized cost model that converts predicted runtime into expected dollars by blending on-demand, reserved, and spot or preemptible pricing, factoring in checkpointing overhead if a spot instance is reclaimed. Fourth, a central scheduler that frames placement as a constrained optimization problem, matching queue depth against available pools while respecting topology, gang scheduling for distributed jobs, and hardware-specific constraints such as Trainium's need for Neuron SDK compilation or TPUs requiring XLA graphs.

COMMON WRONG ANSWERS: The biggest red flag is treating all accelerators as fungible commodity compute. Candidates often propose a simple least-cost-per-hour scheduler without modeling per-step performance, which would place a job on cheap Trainium chips even if the model runs ten times slower than on GPUs and ends up more expensive overall. Another red flag is ignoring migration and checkpoint compatibility; you cannot pause a training job on an NVIDIA GPU and resume it on a Google TPU without converting optimizer states, precision formats, and potentially the entire computational graph. A third red flag is omitting preemption and spot pricing dynamics, which are central to actual cost optimization in the cloud.

LIKELY FOLLOW-UPS: Expect the interviewer to push on how you would handle hardware-specific compilation, how you would update cost models as spot prices fluctuate, or how you would schedule multi-node distributed jobs that require specific network topologies. They may also ask how your abstraction breaks down when a user needs custom CUDA kernels or low-level memory layout control, and whether you expose escape hatches for power users.

ONE CONCRETE EXAMPLE: Suppose a user submits a transformer fine-tuning job. Your platform parses the job spec and sees it needs mixed-precision training with a specific batch size. The performance predictor estimates 0.8 seconds per step on an A100 at 2.50 dollars per hour, 1.1 seconds per step on a Trainium instance at 1.20 dollars per hour, and 0.5 seconds per step on a TPU v4 pod slice at 3.00 dollars per hour. The cost model computes dollars per effective step as 0.00056 for the A100, 0.00037 for Trainium, and 0.00042 for the TPU. The scheduler notices Trainium is currently occupied but has a deep spot GPU pool available at 0.70 dollars per hour, recomputes the GPU cost per step at 0.00016, and places the job there with a checkpointing policy every fifteen minutes in case of preemption.