Design multi-tenant GPU cluster scheduling and preemption policies

WHAT THIS TESTS: This question evaluates whether you can translate high-level multi-tenancy requirements into concrete Kubernetes control plane mechanisms for GPU workloads. The interviewer cares about your understanding of scheduling tradeoffs, isolation boundaries, and preemption semantics at scale, not just listing CRDs. They want to see systems thinking about how quota enforcement, fair sharing, and hardware isolation interact when hundreds of users compete for finite accelerator resources.

A GOOD ANSWER COVERS: First, namespace-level resource governance using ResourceQuotas and LimitRanges to cap GPU count and memory per tenant, paired with cluster-wide fairness layers like Kueue or Volcano that implement hierarchical queues and fair-share scheduling instead of the default kube-scheduler. Second, hardware isolation mechanisms such as NVIDIA MIG for memory and compute partitioning, or time-slicing when full isolation is too expensive, ensuring one tenant cannot cause unpredictable latency for another. Third, preemption policies using PriorityClasses with clear tiering, for example critical inference above training above experimentation, plus graceful preemption hooks that checkpoint model state before eviction to avoid wasted GPU hours. Fourth, observability and enforcement, meaning GPU metrics exported via DCGM feed into autoscalers and quota admission controllers that reject or queue workloads before they hit the scheduler.

COMMON WRONG ANSWERS: A major red flag is proposing static node pools dedicated to each team, which directly contradicts the goal of maximizing utilization and increases infrastructure cost. Another is suggesting the default kube-scheduler with simple node labels will suffice; it lacks fair-share and gang scheduling semantics needed for distributed training jobs. Candidates also err by ignoring memory isolation, assuming GPU count alone protects tenants, when in reality memory bandwidth contention causes the performance interference Red Hat highlights in multitenant environments.

LIKELY FOLLOW-UPS: The interviewer may ask how you handle gang scheduling for multi-node training jobs without head-of-line blocking, or how you would implement preemption without losing hours of GPU work. They might also probe whether you would use virtualization layers like NVIDIA vGPU or SR-IOV instead of Kubernetes-native mechanisms, and when each is appropriate.

ONE CONCRETE EXAMPLE: Imagine a cluster with 100 A100 GPUs. You configure Kueue with cohort-based fair sharing so that no single tenant can consume more than 20% of GPU-hours in a rolling window. Each tenant namespace has a ResourceQuota of 20 GPUs and a LimitRange enforcing MIG profiles. Production inference runs at PriorityClass critical with preemption enabled, while experimental notebooks run at PriorityClass low and can be evicted with a 5 minute grace period. DCGM metrics drive a cluster autoscaler that adds nodes when queue wait times exceed 2 minutes, ensuring utilization stays above 80% without manual intervention.