Data Sharding: Splitting a Database for Scale
Sharding splits a huge database into smaller, independent databases (shards), each on its own server. It's like giving different volumes of a phone book to different librarians. This is critical for massive datasets, but a bad shard key creates 'hot spots'.
Why it exists
A single database server can only get so big or fast. At a certain scale, it becomes a bottleneck, limited by its CPU, memory, or disk I/O. Sharding solves this by allowing a database to scale horizontally, spreading the data and the load across many servers instead of trying to make one server infinitely powerful.
The mental model
Think of sharding as splitting a massive, single-volume encyclopedia into a set of smaller, topic-specific books (A-C, D-F, etc.). Each book is held by a different librarian. To find an entry, you first determine which book it's in (e.g., 'Zebra' is in the T-Z book) and then ask the correct librarian. This is horizontal partitioning—splitting the table by rows. The opposite, vertical partitioning, would be like giving one librarian all the keywords and another all the definitions.
How it works
You choose a 'shard key' from your data, like a user_id or customer_id. A routing function, often a hash function, takes this key and maps it to a specific shard. For example, shard_number = hash(user_id) % number_of_shards. When the application needs to read or write data for a user, it first computes the hash to determine which server to connect to. All data for that user resides on that single shard, making lookups efficient.
When to use it
Sharding is for when you've truly hit the limits of a single machine. It's necessary for applications with massive datasets (terabytes to petabytes), extremely high write throughput, or a need for geographic data distribution to reduce latency for users around the world.
When not to use it
Don't shard prematurely. The complexity is significant. If your dataset fits on a single powerful server, sharding is overkill. It's also a poor fit if your application requires frequent, complex queries that join data across different shard keys. These cross-shard queries are slow, difficult to implement, and negate many of the benefits. Caching or read replicas are often better first steps for read-heavy workloads.
One canonical example
A large e-commerce platform shards its customer order data. It uses customer_id as the shard key. When a customer views their order history, the application hashes their customer_id to find which of the 256 database shards holds their data. This allows the platform to handle millions of concurrent customers and store a virtually unlimited number of orders without a single database becoming a bottleneck.
Interview question
What is a primary challenge when designing an application that utilizes database sharding?
- a.Increased latency for simple data retrieval operations due to the routing function.
- b.Ensuring all data for a single entity is stored across multiple shards for redundancy.
- c.The inability to scale the database horizontally to accommodate growing data volumes.
- d.Managing complex queries that require joining data located on different shards.Correct
Why? this is the answer
The card explicitly states that sharding is a 'poor fit if your application requires frequent, complex queries that join data across different shard keys' because 'these cross-shard queries are slow, difficult to implement, and negate many of the benefits.' Option A is incorrect because individual lookups within a shard are efficient, and sharding aims to reduce overall latency for distributed users.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on database — each one lists the topics its interview covers.
See open roles