Compute & Storage Separation: Scale One Without the Other
This architecture treats your data warehouse (cheap storage) and query engine (expensive compute) as separate services. You can scale compute for peak demand without overprovisioning storage.
Why it exists
Traditional databases bundle storage and compute on a single server. To get more query power, you had to buy a bigger server with more disk space, even if you didn't need the storage. This architecture was invented to break that expensive, inefficient coupling, especially in the cloud where resources are elastic.
The mental model
Think of it as separating a factory from its warehouse. The warehouse (storage) is a massive, cheap building that just holds goods. The factory (compute) is a separate, expensive facility full of machinery that you can rent by the hour. You only pay for factory time when you're making something, and you can add more factory lines without building a new warehouse.
How it works
Compute clusters, running query engines like Snowflake or BigQuery, are fully decoupled from the data storage system. When a query is run, the compute nodes request the necessary data from a centralized, durable object store like Amazon S3 or Google Cloud Storage over the network. The compute cluster processes the data and returns the result. The data itself remains in the storage layer, accessible to other compute clusters.
When to use it
This pattern is the standard for modern cloud data warehousing. Use it for workloads with variable demand, where you need to scale compute resources up or down quickly without affecting storage. It's also ideal for sharing a single source of truth (one data lake) across multiple teams or applications, each with its own dedicated compute resources.
When not to use it
The main drawback is network overhead. For applications requiring extremely low latency and high transaction rates, like an OLTP system, the network round-trip between compute and storage can be a performance killer. In these cases, a traditional, tightly-coupled architecture where data is local to the processor is often superior.
One canonical example
An analytics company stores all its data in Google Cloud Storage. During business hours, they run three separate BigQuery compute clusters for different teams analyzing the shared data. At night, they scale down to one small cluster for batch processing. During a spike in activity, they temporarily add a fourth cluster for a few hours without provisioning any additional storage or duplicating data.
Interview question
What is the primary benefit of decoupling compute from storage in a data warehousing architecture?
- a.It reduces network latency for real-time transactional operations.
- b.It allows for independent scaling of processing power and data capacity.Correct
- c.It simplifies data governance and access control mechanisms.
- d.It ensures data consistency across all analytical workloads.
Why? this is the answer
The card highlights that this architecture was designed to "break that expensive, inefficient coupling" by allowing you to "scale compute for peak demand without overprovisioning storage." Option A is incorrect because the card explicitly states that network overhead makes it unsuitable for applications requiring extremely low latency.
Just read this? Test yourself on what you have been reading.
Read the original → coginiti.co
- #databases
- #architecture
- #cloud
- #scalability
- #cost optimization
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on databases — each one lists the topics its interview covers.
See open roles