How do you diagnose and fix a Kubernetes OOMKilled application?

WHAT THIS TESTS: This question tests whether you understand the boundary between Kubernetes scheduling and kernel-level resource enforcement. Specifically, it probes your knowledge of how memory limits translate to cgroup limits that trigger the OOM killer, how requests influence scheduling and node resource accounting, and whether you can distinguish between an OOMKilled container, a failing liveness probe, and node-pressure eviction. Interviewers also want to see if you understand Quality of Service classes and eviction priority.

A GOOD ANSWER COVERS: First, verify the diagnosis by running kubectl describe pod and checking containerStatuses lastState terminated reason for OOMKilled, plus Events for memory threshold breaches. Second, explain that requests are used by the scheduler to place pods and by the kubelet for node resource tracking, but they do not cap runtime usage; only limits create a hard cgroup boundary enforced by the Linux OOM killer. Third, gather evidence by correlating the memory limit with actual working set using kubectl top pod, container_memory_working_set_bytes from cAdvisor or Prometheus, or memory profiling inside the application. Fourth, distinguish between a legitimate spike requiring more headroom and a memory leak requiring a code fix. Fifth, rightsize resources by setting requests close to the measured baseline p95 usage so the scheduler can bin-pack accurately, then set limits above requests with a safety margin, for example 20 to 40 percent, while aiming for Guaranteed QoS if possible by keeping requests equal to limits for all containers. Sixth, mention node-level protections such as ensuring the node has allocatable memory remaining and that kube-reserved or system-reserved are configured so the node does not enter memory pressure and evict pods before individual containers hit their limits.

COMMON WRONG ANSWERS: A major red flag is saying that requests throttle or limit runtime resource usage; requests are only for scheduling and node accounting. Another is recommending to simply increase limits without profiling, which masks leaks and wastes cluster capacity. Some candidates confuse OOMKilled with eviction; eviction is triggered by node memory pressure and targets pods based on QoS, whereas OOMKilled is container-level and triggered by exceeding the memory limit. Also, suggesting limits lower than requests reveals a fundamental misunderstanding of the API.

LIKELY FOLLOW-UPS: How does the OOM score get calculated and why does the container get killed instead of the pod? What happens if you set a memory limit but no request? How would Vertical Pod Autoscaler change your approach? How do you handle applications with Java heap sizing where the JVM does not see cgroup limits by default? What is the difference between container_memory_usage_bytes and container_memory_working_set_bytes when diagnosing OOM?

ONE CONCRETE EXAMPLE: Imagine a Java service with a 512 Mi memory limit and 256 Mi request. At startup it loads a 400 Mi cache and is immediately OOMKilled. The fix is not to raise the request to 512 Mi alone; you must raise the limit to at least 600 Mi to accommodate the cache plus heap overhead, set the request to 400 Mi to reflect steady state, and configure the JVM to respect cgroup memory limits with -XX:+UseContainerSupport so the heap max stays inside the container boundary rather than trusting physical RAM. If the application is still unstable, you add a VPA recommendation mode to automate future tuning.

Source: kubernetes.io