Durable write path in a sharded KV store

WHAT THIS TESTS Whether you can design a write path that survives node and disk failures while making the durability and latency tradeoffs explicit.

A GOOD ANSWER COVERS The client sends a write for a key. A routing layer or smart client hashes the key, typically with consistent hashing, to locate the responsible shard and its current leader. The leader appends the mutation to its write-ahead log and fsyncs that log entry to durable storage before applying it to the in-memory state, so a crash mid-write can be replayed on restart. The leader then forwards the entry to its two followers. Each follower appends and fsyncs, then acknowledges. With replication factor three and a write quorum of two, the leader confirms success to the client once it plus at least one follower have durably persisted the entry, ensuring the write survives a single node loss.

COMMON WRONG ANSWERS Confirming the client immediately after an in-memory write, skipping the WAL, or assuming asynchronous replication still guarantees durability under a leader crash.

LIKELY FOLLOW-UPS Sync versus async replication and its effect on tail latency; how quorum (W + R greater than N) gives read-your-writes; how leader failover promotes the most up-to-date follower.

ONE CONCRETE EXAMPLE A write to key user:42 hashes to shard 5. Shard 5's leader logs and fsyncs the change, ships it to followers A and B; follower A fsyncs and acks. With W=2 satisfied, the client gets success even though B is briefly behind. If the leader then dies, A, holding the committed entry, is promoted, and no acknowledged write is lost.