Mitigating a database shard hot spot

WHAT THIS TESTS Whether you can separate immediate firefighting from the root-cause fix, and whether you understand that hot spots usually trace to a poor partition key.

A GOOD ANSWER COVERS First diagnose: is it read-heavy or write-heavy, and is one key or a key range responsible? For a read-heavy hot spot, the short-term strategy is to add read replicas to the hot shard and put a cache in front of the popular keys, deflecting most reads before they reach the database; rate limiting an abusive client can also buy time. For the long-term fix, address the partition key itself. If a sequential or timestamp key sends all new writes to one shard, switch to a hashed key so writes spread evenly. For a celebrity key whose single value is overwhelmingly popular, add a salt or random suffix to split it across multiple physical partitions, fanning reads back in at query time. Repartitioning the hot range into finer shards also rebalances load.

COMMON WRONG ANSWERS Only vertically scaling the node, adding replicas without noticing the load is write-heavy (replicas do not absorb writes), or ignoring that the underlying key design will keep recreating the hot spot.

LIKELY FOLLOW-UPS Why hashing hurts range scans; how salting complicates reads; how online resharding avoids downtime.

ONE CONCRETE EXAMPLE An analytics table keyed by event date funnels every write of the current day into one shard. Short term, you cache today's aggregate reads. Long term, you change the key to a hash of event id plus date, scattering today's writes across all shards and eliminating the daily hot spot.