Preventing split-brain in HA Operators

WHAT THIS TESTS Whether you know that controller high availability means redundancy for failover, not parallel reconciliation. The interviewer probes the leader election primitive that controller-runtime and kube-controller-manager rely on.

A GOOD ANSWER COVERS You run several replicas of the Operator, but they elect a single leader and only that pod executes the reconcile loop; the others idle and watch. Election is implemented with a Lease object in the coordination.k8s.io API group. The candidate that successfully writes its identity as holder owns the lease for leaseDurationSeconds and must renew within renewDeadlineSeconds. If it crashes or the network partitions and it stops renewing, the lease expires and a standby acquires it after retryPeriod, becoming the new leader. Because the Lease lives in etcd and updates use optimistic concurrency on resourceVersion, only one writer can win at a time, which is what prevents split-brain.

COMMON WRONG ANSWERS Saying every replica reconciles and conflicts are sorted out by resourceVersion. That prevents conflicting writes to a given object but still wastes work and can cause thrashing or ordering bugs. Another wrong answer is using an external lock like Redis when Kubernetes already provides Leases. Confusing Lease with the older ConfigMap or Endpoints lock is a minor red flag but acceptable as historical context.

LIKELY FOLLOW-UPS What happens during a failover gap when no leader holds the lease? Reconciliation pauses briefly until a standby wins. How do you tune leaseDuration versus renewDeadline to trade failover speed against false failovers under load? How does this differ from StatefulSet ordering guarantees?

ONE CONCRETE EXAMPLE Three Operator pods start; pod-A writes its name into a Lease named my-operator with a fifteen second duration and renews every two seconds. Pods B and C see the lease is held and stay passive. Pod-A's node dies; after the duration elapses without renewal, pod-B's manager acquires the Lease and begins reconciling, so at most one pod ever drives the resource.