Capacity planning for distributed stateful systems

WHAT THIS TESTS: Whether you grasp that stateful, multi-region systems have capacity constraints that go well beyond a single node's CPU, memory, and disk, and that limits must be proven empirically, not assumed.

A GOOD ANSWER COVERS: Beyond raw compute, the dominant factors are: replication overhead and cross-region network bandwidth and latency (every write may be replicated across regions, consuming bandwidth and adding latency that bounds throughput); disk IOPS and throughput, not just capacity, since stateful systems are often I/O bound long before they run out of GB; connection limits, thread pools, and file descriptors, which cap concurrency; partition or shard balance and hotspots, because uneven key distribution can saturate one node while others idle; write amplification and compaction or background work; and consensus or quorum costs in replicated systems. Critically, plan failover headroom: with N regions you must size so that losing one region (N+1, or N+2 for stricter availability) does not overload the survivors, which means running each region below peak. To test limits, use realistic load tests at and beyond projected peak, soak tests to expose leaks and compaction backlogs over time, stress tests to find the breaking point, and chaos experiments that fail an entire region to verify the remaining capacity actually absorbs the traffic. Forecast from historical growth plus known events.

COMMON WRONG ANSWERS: Sizing only on average CPU, memory, and disk; ignoring IOPS, replication bandwidth, and connection limits. Forgetting region-failover headroom, so the system collapses when one region is lost. Trusting theoretical limits without load and chaos testing.

LIKELY FOLLOW-UPS: How do you handle hot partitions? How much headroom for N+1 versus N+2? How do you load test without affecting production?

ONE CONCRETE EXAMPLE: A three-region database is sized so each region normally runs near 60% so that if one region fails, the other two can absorb its traffic without exceeding limits. Load tests confirm IOPS, not CPU, is the ceiling, and a chaos drill killing one region validates the survivors handle the failover within SLO.