Database Sharding: Splitting Data for Scale
Sharding splits a database across multiple servers, like dividing a phone book into A-M and N-Z volumes. It's used when a single server can't handle the data size or write load. The footgun is that querying across shards is complex and slow.
WHY IT EXISTS: A single database server can only get so big or so fast, a process called vertical scaling. When you hit the physical limits of hardware or the cost becomes prohibitive, you need a way to distribute the load across multiple machines. Sharding is the primary method for this horizontal scaling.
THE MENTAL MODEL: Imagine a library with a single, enormous dictionary. It's slow for multiple people to use, and eventually, it will be full. Sharding is like replacing it with multiple volumes (A-F, G-L, etc.), each on its own table. Each volume is a shard—smaller, faster to search, and you can add more volumes as your dictionary grows.
HOW IT WORKS: You choose a "shard key" from your data, like a user_id or region. A function, often based on hashing or ranges, uses this key to decide which shard (which server) a row of data lives on. For example, all users with IDs 1-1000 go to Shard 1, and users 1001-2000 go to Shard 2. The application or a routing proxy directs reads and writes to the correct shard based on the key.
WHEN TO USE IT: Use sharding when your database is bottlenecked by write volume, storage capacity, or CPU on a single machine. It's essential for applications with massive datasets or extremely high transaction rates, like large-scale social media platforms or e-commerce sites.
WHEN NOT TO USE IT: Avoid sharding for small to medium-sized applications where a single powerful server is sufficient and far less complex. It's a poor fit if your queries frequently require joining data that would end up on different shards, as this negates the performance benefits and adds significant complexity.
ONE CANONICAL EXAMPLE: A ride-sharing app shards its data by geographic region. All trips, drivers, and users in North America are on one set of shards, while data for Europe is on another. This keeps queries fast for most operations (e.g., finding a nearby driver) because the data is co-located. A query for "all trips globally" would be very slow and complex.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.