Snowflake: Decoupled Storage and Compute

Why it exists

Traditional data warehouses often tightly coupled compute power and data storage. To get more query performance, you had to scale up the entire system, including storage, which was inefficient and expensive. Snowflake was built for the cloud to solve this by separating these two concerns entirely.

The mental model

Think of Snowflake as a library with a massive, central book warehouse (storage) and many independent reading rooms (compute). Any reader can access any book. If a large class needs to study, you can open a new, large reading room just for them without expanding the warehouse. When they're done, you close the room to save on heating and lighting costs. The warehouse is your data; the reading rooms are your 'virtual warehouses' for queries.

How it works

Snowflake's architecture is a hybrid of shared-disk and shared-nothing designs. It has three key layers. First, a central Database Storage layer holds all data, automatically optimized into a compressed, columnar format called micro-partitions. Second, a Compute layer uses 'virtual warehouses'—massively parallel processing (MPP) clusters—to run queries. These are independent of storage and each other. Third, a Cloud Services layer acts as the brain, managing transactions, security, and query optimization. This lets multiple teams query the same data with different-sized warehouses without competing for resources.

When to use it

Use Snowflake for cloud-based data warehousing and analytics, especially with variable workloads. It excels when you need to run large, complex queries for a short time, or when multiple teams need to query the same data without performance interference. It's also strong for building data platforms that unify structured, semi-structured (JSON), and unstructured data.

When not to use it

Snowflake is not primarily designed for high-throughput online transaction processing (OLTP) like a traditional relational database, though its Hybrid Tables feature is addressing this. As a managed service, you cannot run it on-premises or in a private cloud, making it unsuitable for environments with strict data residency rules that forbid public clouds.

One canonical example

A retail company stores years of sales data in Snowflake. The marketing team runs a large virtual warehouse for a few hours to analyze customer behavior for a new campaign. Simultaneously, the finance team uses a smaller, always-on warehouse for routine reporting. Both teams query the same central data without impacting each other. When the marketing analysis is done, its warehouse is suspended, stopping all compute costs.