Preventing split-brain in HA Operators
leader election in controllers.
run active-passive replicas, only the leader reconciles, election uses a Lease object renewed under a TTL.
thinking all replicas reconcile in parallel or relying on optimistic locking alone.
What's really being asked
Whether you know that controller high availability means redundancy for failover, not parallel reconciliation. The interviewer probes the leader election primitive that controller-runtime and kube-controller-manager rely on.
The full answer
You run several replicas of the Operator, but they elect a single leader and only that pod executes the reconcile loop; the others idle and watch. Election is implemented with a Lease object in the coordination.k8s.io API group. The candidate that successfully writes its identity as holder owns the lease for leaseDurationSeconds and must renew within renewDeadlineSeconds. If it crashes or the network partitions and it stops renewing, the lease expires and a standby acquires it after retryPeriod, becoming the new leader. Because the Lease lives in etcd and updates use optimistic concurrency on resourceVersion, only one writer can win at a time, which is what prevents split-brain.
The mistakes people make
Saying every replica reconciles and conflicts are sorted out by resourceVersion. That prevents conflicting writes to a given object but still wastes work and can cause thrashing or ordering bugs. Another wrong answer is using an external lock like Redis when Kubernetes already provides Leases. Confusing Lease with the older ConfigMap or Endpoints lock is a minor red flag but acceptable as historical context.
What usually comes next
What happens during a failover gap when no leader holds the lease? Reconciliation pauses briefly until a standby wins. How do you tune leaseDuration versus renewDeadline to trade failover speed against false failovers under load? How does this differ from StatefulSet ordering guarantees?
A concrete example
Three Operator pods start; pod-A writes its name into a Lease named my-operator with a fifteen second duration and renews every two seconds. Pods B and C see the lease is held and stay passive. Pod-A's node dies; after the duration elapses without renewal, pod-B's manager acquires the Lease and begins reconciling, so at most one pod ever drives the resource.
Interview question
You scale your Operator to three replicas for availability. What is the expected reconciliation behavior across those pods at steady state?
- a.All three reconcile in parallel, with resourceVersion preventing conflicts
- b.Each pod reconciles a sharded third of the custom resources
- c.Reconciliation is disabled until you set replicas back to one
- d.Only the pod holding the Lease reconciles; the others stand byCorrect
Why? this is the answer
HA Operators are active-passive: one leader holds a coordination Lease and reconciles while others wait to take over. Parallel reconciliation wastes work and risks thrashing even though resourceVersion blocks conflicting writes.
Just read this? Test yourself on what you have been reading.
Read the original → kubernetes.io
- #kubernetes
- #operators
- #leader-election
- #leases
- #high-availability
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on kubernetes — each one lists the topics its interview covers.
See open roles