How does a StatefulSet recover a Pod after node failure?

WHAT THIS TESTS Whether you understand the StatefulSet recovery sequence, the at-most-one-Pod-per-ordinal safety guarantee, and how volumeClaimTemplates preserve data.

A GOOD ANSWER COVERS When the node hosting a Pod fails, the node controller marks the node NotReady after missing heartbeats, and after a timeout the Pods on it are slated for deletion. For a StatefulSet, the controller maintains at most one Pod per ordinal at a time, so it will not immediately spin up a replacement while the old Pod's true state is unknown, since two live writers on the same database volume would corrupt data. Recovery proceeds once the old Pod is confirmed gone, either because the node is deleted, the kubelet recovers and confirms termination, or an operator force-deletes the Pod. The replacement Pod is created with the same ordinal identity and the same stable DNS name. Crucially, volumeClaimTemplates created a dedicated PVC bound to that ordinal, and the new Pod reattaches that exact PVC. For block storage the disk detaches from the dead node and attaches to the new node, so the database resumes with its existing on-disk state intact rather than empty.

COMMON WRONG ANSWERS Saying recovery is instant and fully automatic even when the node is merely unreachable. Believing the new Pod gets fresh empty storage. Ignoring the split-brain risk that drives the cautious behavior. Force-deleting Pods without understanding the danger.

LIKELY FOLLOW-UPS Why will Kubernetes not auto-replace a Pod on an unreachable node? When is force deletion appropriate? How does the volume detach and reattach across nodes? How does the database layer handle catch-up replication?

ONE CONCRETE EXAMPLE Node hosting cassandra-2 dies; the node goes NotReady, the Pod cannot be confirmed dead, so the operator deletes the node object; cassandra-2 is recreated on a healthy node, its EBS PVC from volumeClaimTemplates detaches from the dead node and reattaches, and Cassandra resumes with its prior SSTables and catches up via replication.