Propose an architectural solution for contended GPU training resources

WHAT THIS TESTS: The interviewer wants to see if you understand GPU clusters as a constrained expensive resource that requires multi-tenant scheduling fault-tolerant design and cost optimization. They care about your ability to bridge Kubernetes scheduling primitives with ML workload patterns like gang scheduling checkpointing and heterogeneous job types.

A GOOD ANSWER COVERS: First workload segmentation. Separate training from inference critical from best-effort and checkpoint-friendly from non-checkpointable jobs. Second quota and priority mechanisms. Use namespace-level GPU quotas and Kubernetes PriorityClasses with preemption so high-priority jobs can displace low-priority ones without starving teams. Third spot and preemptible integration. Route fault-tolerant training to GPU spot instances with automated checkpointing to object storage and keep on-demand GPUs for interactive or deadline-sensitive workloads. Fourth finer-grained sharing. Deploy Multi-Instance GPU or time-slicing for inference and small-model tasks to increase bin-packing density. Fifth scheduler enhancements. Replace the default scheduler with a GPU-aware scheduler that supports gang scheduling for distributed training topology-aware placement for NVLink and fragmentation-aware bin-packing.

COMMON WRONG ANSWERS: Suggesting simply adding more GPU nodes without fixing the scheduling logic. Ignoring preemption and claiming quotas alone solve contention. Treating all workloads identically such as putting interactive notebooks and multi-day training jobs in the same queue. Proposing spot instances without a checkpointing or retry strategy which leads to wasted compute and angry users. Overlooking inference workloads that could share GPUs via MIG while leaving training jobs on full cards.

LIKELY FOLLOW-UPS: How do you handle gang scheduling when only half the requested GPUs are available? What is your strategy for topology-aware placement in multi-node training? How do you price back spot instance interruptions to users? How would you schedule workloads across different GPU generations like A100 versus H100? What observability metrics would you expose to debug queue latency?

ONE CONCRETE EXAMPLE: A platform team runs a Kubernetes cluster with NVIDIA A100s. Data scientists submit long-running fine-tuning jobs that can checkpoint every epoch while model serving pods need only two gigabytes of GPU memory. The team installs the NVIDIA GPU Operator and enables MIG to partition each A100 into smaller instances for serving. They configure two PriorityClasses so production-training overrides best-effort-training. Namespace quotas cap each team at eight full GPUs. Spot instances handle best-effort training with a gang scheduler that preempts low-priority jobs and checkpoints state to object storage before eviction. Queue times for production jobs drop by sixty percent and overall cluster utilization rises from fifty-five to eighty-five percent.

Source: kubezilla.io