etcd: Kubernetes's Single Source of Truth

WHY IT EXISTS: Kubernetes needs a reliable, consistent place to store the state of the entire cluster. Without a central source of truth, different components like the scheduler and controller manager could have conflicting views of what should be running, leading to chaos. etcd was chosen to solve this by providing a distributed, fault-tolerant data store specifically for this purpose.

THE MENTAL MODEL: Think of etcd as the definitive logbook for a ship called "Kubernetes." The captain (the API server) is the only one allowed to write in it. Every other officer (scheduler, controller manager, kubelet) asks the captain what the logbook says. It records the desired state ("we should have 3 lookout pods") and the observed state, ensuring everyone works from the same plan.

HOW IT WORKS: etcd is a distributed key-value store that uses the Raft consensus algorithm. A cluster of etcd nodes (typically an odd number like 3 or 5 for high availability) elects a leader. All write operations go through the leader, which then replicates the changes to its followers. A write is only committed after a majority of nodes acknowledge it. This guarantees that even if some nodes fail, the cluster state remains consistent and available as long as a majority (a quorum) is online. The Kubernetes API server is the primary client that reads from and writes to etcd.

WHEN TO USE IT: etcd is the backbone of every Kubernetes control plane. It's used implicitly every time you run kubectl apply or kubectl get. The API server translates your request into a read or write operation against etcd. It's essential for storing all cluster objects: Pods, Deployments, Services, Secrets, ConfigMaps, and custom resources. You also interact with it indirectly when setting up a high-availability Kubernetes cluster.

WHEN NOT TO USE IT: Never store application data directly in etcd. It's not designed for high-volume writes or large data objects; use a proper database for your application's state. Also, avoid interacting with etcd directly unless you are performing cluster administration or disaster recovery. Always go through the Kubernetes API server for normal operations to ensure validation and admission control.

ONE CANONICAL EXAMPLE: A user runs kubectl scale deployment my-app --replicas=5. The API server receives this request and writes the new desired replica count for the my-app Deployment object into etcd. The controller manager, which is constantly watching for changes via the API server, sees the discrepancy between the desired state (5) and the current state (e.g., 3). It then creates two new Pods to match the desired state. etcd stored the initial change that triggered this entire reconciliation loop.