Propose an architectural solution for contended GPU training resources
Tests multi-tenant GPU scheduling design at scale. Great answers tier jobs by checkpointability, apply quota-based preemption, mix spot and on-demand instances, and use MIG or time-slicing to bin-pack. Red flag: buying GPUs without scheduling logic.
What's really being asked
The interviewer wants to see if you understand GPU clusters as a constrained expensive resource that requires multi-tenant scheduling fault-tolerant design and cost optimization. They care about your ability to bridge Kubernetes scheduling primitives with ML workload patterns like gang scheduling checkpointing and heterogeneous job types.
The full answer
First workload segmentation. Separate training from inference critical from best-effort and checkpoint-friendly from non-checkpointable jobs. Second quota and priority mechanisms. Use namespace-level GPU quotas and Kubernetes PriorityClasses with preemption so high-priority jobs can displace low-priority ones without starving teams. Third spot and preemptible integration. Route fault-tolerant training to GPU spot instances with automated checkpointing to object storage and keep on-demand GPUs for interactive or deadline-sensitive workloads. Fourth finer-grained sharing. Deploy Multi-Instance GPU or time-slicing for inference and small-model tasks to increase bin-packing density. Fifth scheduler enhancements. Replace the default scheduler with a GPU-aware scheduler that supports gang scheduling for distributed training topology-aware placement for NVLink and fragmentation-aware bin-packing.
The mistakes people make
Suggesting simply adding more GPU nodes without fixing the scheduling logic. Ignoring preemption and claiming quotas alone solve contention. Treating all workloads identically such as putting interactive notebooks and multi-day training jobs in the same queue. Proposing spot instances without a checkpointing or retry strategy which leads to wasted compute and angry users. Overlooking inference workloads that could share GPUs via MIG while leaving training jobs on full cards.
What usually comes next
How do you handle gang scheduling when only half the requested GPUs are available? What is your strategy for topology-aware placement in multi-node training? How do you price back spot instance interruptions to users? How would you schedule workloads across different GPU generations like A100 versus H100? What observability metrics would you expose to debug queue latency?
A concrete example
A platform team runs a Kubernetes cluster with NVIDIA A100s. Data scientists submit long-running fine-tuning jobs that can checkpoint every epoch while model serving pods need only two gigabytes of GPU memory. The team installs the NVIDIA GPU Operator and enables MIG to partition each A100 into smaller instances for serving. They configure two PriorityClasses so production-training overrides best-effort-training. Namespace quotas cap each team at eight full GPUs. Spot instances handle best-effort training with a gang scheduler that preempts low-priority jobs and checkpoints state to object storage before eviction. Queue times for production jobs drop by sixty percent and overall cluster utilization rises from fifty-five to eighty-five percent.
Interview question
Which combination of strategies best addresses GPU resource contention when a platform team must support both long-running distributed training and low-latency model serving on the same A100 cluster?
- a.Segmenting workloads by checkpointability and criticality, using PriorityClasses with preemption for training tiers, and reserving MIG slices for serving while keeping full GPUs for distributed trainingCorrect
- b.Routing all training jobs to GPU spot instances without checkpointing logic and using the default Kubernetes scheduler with topology-unaware placement
- c.Placing all workloads into a single queue with namespace quotas and scaling the cluster horizontally whenever GPU pending pods exceed a threshold
- d.Enabling Multi-Instance GPU for all workloads and disabling preemption to avoid interrupting training jobs
Why? this is the answer
The correct approach segments workloads by checkpointability and criticality, applies PriorityClasses with preemption for training, and uses MIG only for serving to maximize utilization. Option C is tempting because horizontal scaling and quotas seem like straightforward fixes, but the card explicitly flags buying GPUs without scheduling logic and relying solely on quotas as inadequate solutions that ignore workload heterogeneity.
Just read this? Test yourself on what you have been reading.
Read the original → kubezilla.io
- #mlops
- #kubernetes
- #gpu-scheduling
- #distributed-systems
- #resource-management
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on mlops — each one lists the topics its interview covers.
See open roles