Kubernetes Taints and Tolerations: Repelling Pods

WHY IT EXISTS To give cluster administrators control over which pods can run on which nodes. Without this, the scheduler might place a general-purpose web server on an expensive GPU node, wasting resources. Taints provide a mechanism to reserve nodes for specific workloads by repelling pods that are not explicitly allowed.

THE MENTAL MODEL Think of a taint as a "No Trespassing" sign you put on a node. By default, no pod can enter. A toleration is a special key that a pod carries, allowing it to ignore a specific "No Trespassing" sign and get scheduled on that node. It's a gatekeeping mechanism, not an invitation.

HOW IT WORKS You apply a taint to a node using a key, a value, and an effect. The three effects are: NoSchedule (new pods without a matching toleration won't be scheduled), PreferNoSchedule (the scheduler will try to avoid placing pods without a toleration), and NoExecute (any running pods without a matching toleration will be evicted). A Pod's spec must include a toleration that matches the taint's key, value, and effect to be placed on the node.

WHEN TO USE IT Use taints and tolerations to dedicate nodes for specific purposes. This is common for nodes with special hardware like GPUs, for isolating critical monitoring or logging daemons, or for ensuring that only certain applications can run on nodes with higher security requirements. Kubernetes itself uses taints for node conditions like being unreachable or out of disk space.

WHEN NOT TO USE IT Do not use taints and tolerations as your primary mechanism for attracting pods to a set of nodes. While they are often used with node affinity to achieve this, their main purpose is repulsion. If you just want to express a preference for a node, nodeAffinity is the correct tool. Using only taints can lead to complex scheduling logic.

ONE CANONICAL EXAMPLE To reserve a node for GPU workloads, you would taint the node: kubectl taint nodes node1 gpu=true:NoSchedule. Then, in your machine learning pod's YAML definition, you would add a toleration: tolerations: - key: "gpu" operator: "Equal" value: "true" effect: "NoSchedule". Now, only pods with this specific toleration can be scheduled on node1.