Design training job submission to a shared Kubernetes cluster

What's really being asked

This question evaluates whether you can bridge raw Kubernetes primitives with ML-specific operational needs. The interviewer wants to see multi-tenant design instincts, awareness of batch scheduling nuances for GPUs, and a production-grade observability story. You are expected to move beyond a simple pod spec and discuss lifecycle management, dependency isolation, and fair resource reclamation in a shared cluster.

The full answer

Four layers in order. First, a submission abstraction such as a CLI, SDK, or web portal that translates data scientist code into Kubernetes custom resources like a TrainJob or Pipeline, hiding YAML complexity. Second, dependency management through versioned container images, artifact stores for datasets and models, and possibly an image cache or PVC strategy to avoid repeated large pulls. Third, resource allocation via namespace-level quotas, dedicated GPU node pools, taints and tolerations, and an advanced scheduler such as Volcano, Kueue, or YuniKorn to enable gang scheduling and prevent GPU fragmentation. Fourth, monitoring and cost control using Prometheus for metrics, a logging backend for distributed training logs, and per-namespace or per-user cost attribution to charge back resource usage.

The mistakes people make

Treating training jobs as stateless microservices is the biggest red flag. Answers that rely solely on the default Kubernetes scheduler ignore gang scheduling requirements for distributed training, leading to deadlock when only half of an eight-GPU job lands. Proposing one giant shared PVC for all datasets introduces noisy neighbor issues and security risks. Forgetting checkpoint persistence means data scientists lose days of work on preemption. Ignoring queueing entirely leads to cluster congestion and priority inversion.

What usually comes next

How would you handle preemption and resume a distributed training job that loses a node? What changes if the cluster spans multiple availability zones or on-prem and cloud? How do you schedule interactive notebook pods differently from batch training jobs? What is your strategy for multi-tenancy when one team submits a thousand hyperparameter trials via Katib?

A concrete example

Suppose a team submits a PyTorch distributed job requiring eight GPUs on four nodes. Your gateway writes a TrainJob custom resource. Kueue or Volcano queues the job until the full set of GPUs is available, preventing partial allocation. The job pulls a cached image from a local registry, mounts read-only training data from an object store via a CSI driver, and writes checkpoints to a distributed filesystem every fifteen minutes. Prometheus scrapes GPU utilization and memory metrics, while Loki aggregates logs from all four workers. If the job exceeds its twelve-hour quota or a higher-priority job arrives, the scheduler preempts it, the training framework resumes from the latest checkpoint, and the cost attribution system bills the team for ninety-two GPU-hours consumed.