Apache ZooKeeper: A Coordinator for Distributed Systems

WHY IT EXISTS: Building distributed systems is hard because nodes need to agree on state. Who is the leader? What is the current configuration? Where are the other services? ZooKeeper was built to provide a robust, off-the-shelf solution for this coordination problem, so every application doesn't have to reinvent consensus.

THE MENTAL MODEL: ZooKeeper is like a special-purpose, highly-available file system for your cluster's metadata. You don't store large application data in it. Instead, you store small, critical pieces of information (called znodes) that all your services need to see consistently, like a configuration flag or the address of the current primary database. It's a source of truth.

HOW IT WORKS: A ZooKeeper cluster is called an ensemble, typically of 3 or 5 servers. To ensure reliability, data is replicated across the ensemble. Writes go through a leader and are committed only after a majority of servers acknowledge them, using a consensus protocol called ZAB (ZooKeeper Atomic Broadcast). Clients maintain a persistent session and can set "watches" on data nodes to be notified of changes.

WHEN TO USE IT: It's a foundational component in many distributed data systems. Use it for service discovery (finding other services), leader election (deciding which node is in charge), and distributed configuration management. Systems like Apache Kafka, HBase, and Solr historically relied heavily on it.

WHEN NOT TO USE IT: Don't use it as a general-purpose database. It's designed for low-volume, small-payload metadata, not high-throughput application data. For new projects, consider lighter-weight alternatives like etcd or Consul, as ZooKeeper is known for being operationally complex to manage. Many modern systems are also removing their ZooKeeper dependency.

ONE CANONICAL EXAMPLE: Apache Kafka brokers historically used ZooKeeper to elect a "controller" broker. The first broker to create a special, temporary znode /controller becomes the leader. If that broker fails, its session expires, the znode is automatically deleted, and other brokers race to create it again, ensuring a new controller is always elected.