Iceberg vs Delta Lake metadata and ACID

WHAT THIS TESTS This evaluates deep familiarity with how open table formats layer ACID transactions, snapshot isolation, and schema evolution over immutable object storage.

METADATA STRUCTURE Apache Iceberg maintains a tree. A metadata file describes the table and points to the current snapshot; each snapshot points to a manifest list, which points to manifest files, which list the actual data files with column-level statistics. A commit creates new metadata and atomically swaps the pointer to the latest metadata file, often via a catalog. Delta Lake uses a single ordered transaction log under a _delta_log directory: each commit is a numbered JSON file recording added and removed files and metadata, with periodic Parquet checkpoints to compact the log for fast reads.

ACID AND CONCURRENT WRITES Both use optimistic concurrency control rather than locks, which suits immutable object stores. A writer reads the current snapshot, prepares its changes, then attempts to commit by advancing to the next version; if another writer committed first, it detects the conflict and either retries on the new snapshot or fails, depending on whether the changes are compatible. The atomic step is creating the next log entry (Delta) or swapping the metadata pointer (Iceberg). Both deliver snapshot isolation and time travel to prior versions.

SCHEMA EVOLUTION Both support adding, renaming, reordering, and dropping columns as metadata-only operations without rewriting data. Iceberg tracks columns by a stable internal field id, so renames are safe and do not depend on position or name, a notably robust design. Delta records schema changes in the log and supports evolution and enforcement, with column mapping for safe renames.

LIKELY FOLLOW-UPS What happens on a write conflict. How does time travel work. Why are field ids safer for renames. How do checkpoints speed up Delta reads.

ONE CONCRETE EXAMPLE Two jobs append to the same Iceberg table. Both read snapshot N; the first commits snapshot N+1 by swapping the metadata pointer. The second's commit fails the optimistic check, so it re-reads N+1 and retries its append on top, producing N+2 without losing either write.