Data Partitioning: Spreading Data for Scalability
Partitioning splits a huge dataset across many machines, like assigning phonebook sections to different librarians. This allows systems to scale beyond a single server.
Why it exists
A single server has finite disk space, memory, and processing power. To handle datasets or query loads that exceed these limits, systems must scale horizontally by adding more machines. Partitioning is the core strategy for distributing a single logical dataset across a cluster of physical machines.
The mental model
Think of a single, massive phonebook and one librarian. Finding a number is slow. Partitioning is like splitting that phonebook into 26 volumes (A, B, C...) and hiring 26 librarians, one for each volume. Now, requests can be handled in parallel, dramatically increasing throughput. In this analogy, the volumes are partitions (or shards), and the librarians are nodes (servers). The goal is to spread both the data and the query load evenly.
How it works
There are two primary strategies for partitioning key-value data: Partitioning by Key Range: Each partition is assigned a continuous range of keys (e.g., Partition 1 holds keys A-D, Partition 2 holds E-G). This method is very efficient for range scans, like fetching all users with names starting with 'S'. Partitioning by Hash of Key: A hash function is applied to the key, and the resulting hash value determines which partition the data belongs to. A good hash function distributes keys uniformly, even if the original keys are not evenly distributed. This helps avoid hot spots.
When to use it
Use partitioning when your data or query volume is too large for a single machine to handle effectively. It is a foundational concept for most large-scale distributed databases, including MongoDB, Elasticsearch, Cassandra, and Bigtable. The choice between range or hash partitioning depends on your application's primary query patterns. If you do many range queries, range partitioning is better. If you primarily do single-key lookups and want to ensure an even load, hash partitioning is superior.
When not to use it
Avoid partitioning if your dataset can comfortably fit on a single, powerful server (vertical scaling). Partitioning introduces significant operational complexity in managing data distribution, routing queries to the correct node, and rebalancing partitions as the cluster grows or shrinks. Also, do not confuse partitioning with replication. Replication copies data for fault tolerance, while partitioning splits data for scalability. They are often used together but solve different problems.
One canonical example
An application partitions user data by a timestamp key using key-range partitioning. All new signups and activity logs are written with the current time. This creates a severe hot spot: all write traffic is directed to the single partition responsible for the current time range, overwhelming that node while all other nodes sit idle. A better strategy would be to partition by a hash of the user ID, which would spread writes evenly across all partitions regardless of when they occur.
Interview question
A database system uses key-range partitioning. Which scenario is most likely to cause a performance bottleneck due to a "hot spot"?
- a.Distributing data where keys are evenly spread across the entire possible range.
- b.Performing single-key lookups that require checking multiple partitions.
- c.Storing new records whose keys are primarily sequential, like timestamps or auto-incrementing IDs.Correct
- d.Executing queries that retrieve data within a specific key range.
Why? this is the answer
The card's canonical example highlights that key-range partitioning with sequential keys (e.g., timestamps) can create a hot spot by directing all new write traffic to a single partition. Option D describes a strength of key-range partitioning, not a bottleneck. Options B describes an ideal scenario, and option B describes a general inefficiency, not the specific hot spot issue for sequential keys.
Just read this? Test yourself on what you have been reading.
Read the original → timilearning.com
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on distributed systems — each one lists the topics its interview covers.
See open roles