Pod Priority: Deciding Who Gets Evicted in Kubernetes

Why it exists

In a busy Kubernetes cluster, not all workloads are equal. A non-essential batch job could occupy resources needed by a critical API server. Without a way to signal importance, the scheduler might make poor trade-offs. Pod Priority and Preemption were created to solve this by letting you define a clear hierarchy for resource contention.

The mental model

Think of it like airline boarding groups. Every pod is a passenger, but some have a Group 1 ticket (high priority) and others have Group 5 (low priority). When the plane (a node) is full, a Group 1 passenger can still get on. The airline will bump a standby passenger (a low-priority pod) to make room. The PriorityClass object is the boarding pass that shows your group number.

How it works

You first create a cluster-level PriorityClass object, which contains a name and an integer value. Higher values mean higher priority. You then assign this class to a pod via the priorityClassName field in its spec. When a high-priority pod is pending and there's no space, the scheduler identifies a node with lower-priority pods. It then preempts (evicts) one or more of them to make room for the more important pod. The evicted pods are terminated gracefully.

When to use it

Use it to protect critical system components like CoreDNS, CNI plugins, or your monitoring stack. It's also vital for ensuring your most important user-facing applications have precedence over less critical background jobs. In a shared cluster, you can use it to give production workloads priority over staging or development workloads.

When not to use it

Avoid applying priority to every pod. If everything is high-priority, the system becomes meaningless—this is called 'priority inflation'. Don't use it as a substitute for proper capacity planning or autoscaling. If your cluster is constantly preempting pods, it's a sign that it's chronically under-provisioned and you should add more nodes.

One canonical example

A common pattern is creating a system-cluster-critical PriorityClass with a very high value (e.g., 2,000,000,000). This class is then applied to essential Kubernetes add-ons. This ensures that even if a user fills the cluster with low-priority pods, core services can always be scheduled and will not be evicted, preventing a total cluster meltdown.