Apache Iceberg: A Table Format for Huge Datasets

Why it exists

Traditional data lakes, often based on Hive, treat file directories as table partitions. This is brittle. Updating data across thousands of files is slow and not atomic, leading to inconsistent reads where different users see different versions of the data. Schema changes often require rewriting the entire table, which is expensive and error-prone.

The mental model

Apache Iceberg is an open table format specification, not a storage engine or a query tool. It's like adding a transactional manifest to your data files in object storage (like S3). Instead of pointing a query engine at a directory of files, you point it at an Iceberg table. The Iceberg metadata layer then tells the engine exactly which files, and which versions of those files, make up the current, consistent state of the table.

How it works

Iceberg maintains a history of the table using snapshots. A top-level metadata file points to the current snapshot. Each snapshot is a manifest list, which points to manifest files. Each manifest file contains a list of the actual data files (e.g., Parquet, ORC) that belong to that version of the table, along with file-level statistics. When you update the table, Iceberg creates new data files and a new snapshot pointing to the new set of files, leaving old ones untouched. This enables atomic commits, reliable schema evolution, and time-travel queries.

When to use it

Use Iceberg when multiple engines (like Spark, Flink, Trino) or teams need to concurrently and reliably access a huge, evolving dataset in a data lake. It's ideal for scenarios requiring ACID-like transactions, safe schema evolution (e.g., adding/renaming columns without rewriting old data), and efficient queries that avoid scanning unnecessary files or partitions.

When not to use it

For small, single-engine workloads or traditional OLTP databases, Iceberg is overkill. It adds a layer of metadata management that isn't necessary if you don't face the consistency, performance, and concurrency challenges of a large-scale, multi-engine data lake.

One canonical example

A streaming pipeline using Flink writes data into an Iceberg table on S3. Simultaneously, a data science team uses Spark to read that same table for model training, while analysts use Trino for ad-hoc queries. Iceberg ensures all three engines see a consistent view of the data, even as it's being continuously updated, without read/write conflicts.