Design a secure multi-tenant CI/CD runner on Kubernetes

WHAT THIS TESTS: This question evaluates whether you can design a hard multi-tenant Kubernetes environment for inherently untrusted CI workloads. Interviewers want to see layered defense covering cluster-level controls, node-level boundaries, and runtime sandboxing. They also care about operational fairness and cost control when multiple teams share compute.

A GOOD ANSWER COVERS: Four layers in order. First, namespace isolation with strict NetworkPolicies that default-deny cross-tenant traffic and only allow egress to required registries and artifact stores. Second, Pod Security Standards enforcing restricted profiles via Pod Security Admission so builds cannot run privileged containers or mount host paths. Third, resource governance using per-namespace ResourceQuotas for aggregate CPU and memory and LimitRanges to set default and max requests per pod, preventing one team from exhausting cluster capacity. Fourth, node-level separation using taints and tolerations or dedicated node pools so sensitive tenants do not share kernels with arbitrary build code, supplemented by sandboxed runtimes like gVisor or Kata Containers where supply-chain risk is high.

COMMON WRONG ANSWERS: Red flags include relying solely on RBAC and labels for isolation, since compromised pods bypass both. Another mistake is omitting LimitRanges, which allows a single build pod to request 64 cores and starve neighbors. Suggesting privileged sidecars for Docker-in-Docker without mentioning rootless or sandboxed alternatives is also dangerous. Finally, ignoring admission control means tenants can accidentally or maliciously violate security policies.

LIKELY FOLLOW-UPS: How would you handle image pull secrets without leaking them across namespaces? What happens when a tenant legitimately needs privileged builds for kernel module compilation? How do you charge back cloud costs per team? Would you use virtual clusters or cluster-per-tenant instead, and at what fleet size?

ONE CONCRETE EXAMPLE: A platform team runs GitLab CI on a 200-node EKS cluster. Each engineering team gets a namespace with a ResourceQuota of 100 CPU and 200 Gi memory and a LimitRange capping any single pod at 8 CPU. NetworkPolicies block all ingress and only allow egress to Artifactory and AWS APIs. Build pods land on a dedicated node pool tainted ci-workloads via tolerations, and pods from external contractors additionally use Kata Containers. Pod Security Admission rejects any pod requesting hostNetwork or runAsRoot. This setup prevented a crypto-mining build from escaping its namespace and capped blast radius during a supply-chain incident last quarter.

Source: kubernetes.io