Impact of losing etcd quorum
control vs data plane separation.
without quorum etcd goes read-only-ish and the API server cannot persist writes, so scheduling and changes stall, but kubelets keep running existing pods.
What's really being asked
Whether you understand that Kubernetes separates the control plane, which manages desired state in etcd, from the data plane, which runs workloads, and that they degrade independently.
The full answer
etcd is a Raft-based store that needs a majority quorum to elect a leader and commit writes. Lose quorum, for example two of three members down, and no writes can be committed. The control-plane impact is immediate: the kube-apiserver can no longer persist changes, so creating, updating, scaling or deleting objects fails, deployments and rollouts stall, the scheduler cannot record bindings, and controllers cannot reconcile. Reads may still partly serve from cache or a lingering member but are unreliable, so treat the cluster as frozen for changes. The data-plane impact is far gentler: kubelets on worker nodes keep running the pods already scheduled to them because they operate from their last-known local spec, containers keep serving traffic, and kube-proxy keeps applying existing service routing rules. What you lose is self-healing: a crashed pod will not be rescheduled, a failed node's pods will not move, autoscaling stops, and new endpoints are not reflected. So existing traffic flows but the cluster cannot adapt.
The mistakes people make
Saying all workloads crash immediately; running pods persist. Claiming reads work perfectly; without quorum consistency guarantees are lost. Forgetting that recovery, rescheduling and scaling are blocked even though current pods survive. Conflating an apiserver outage with workload outage.
What usually comes next
Why three or five members and never an even number? How do you recover from quorum loss, restore from snapshot or remove failed members? What happens if a node fails while etcd is down? How long can the data plane coast, and what eventually breaks?
A concrete example
A three-member etcd loses two members. A web app already running keeps serving users and its Service keeps routing. But when one of its pods OOMKills, it is not recreated because the controller cannot write to etcd, and a kubectl scale command hangs and fails. Once quorum is restored from the surviving member or a snapshot, the API server resumes writes and the missing pod is finally rescheduled.
Interview question
Your three-member etcd loses quorum at 2 a.m. Your customer-facing app was already running. What is the most accurate immediate effect?
- a.Existing pods keep serving traffic, but no new scheduling, scaling, or self-healing can occurCorrect
- b.The data plane stops but the control plane continues accepting writes
- c.Nothing changes because etcd only matters during cluster bootstrap
- d.All running pods crash instantly because etcd holds their state
Why? this is the answer
Kubelets run already-scheduled pods from local state, so traffic continues, but without quorum the API server cannot persist writes, freezing scheduling, scaling and self-healing. Running workloads do not crash merely because etcd lost quorum.
Just read this? Test yourself on what you have been reading.
Read the original → labitlearnit.com
- #kubernetes
- #etcd
- #control-plane
- #quorum
- #reliability
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on kubernetes — each one lists the topics its interview covers.
See open roles