Schema evolution without rewriting history
schema evolution strategy.
use a table format with metadata-level evolution, add a new column rather than mutating the old, and reconcile types at read time; avoid rewriting petabytes.
What's really being asked
This evaluates whether you can manage breaking schema changes over petabyte-scale history using metadata-aware techniques instead of brute-force rewrites that are slow, expensive, and risky.
The full answer
The goal is to absorb the source's change while leaving the existing data files untouched and downstream consumers working. Modern table formats are the key enabler: Apache Iceberg and Delta Lake track schema using stable column identifiers and a metadata layer, so operations like adding a column, renaming, or widening a compatible type are metadata-only and do not rewrite data. For a genuinely incompatible type change, prefer additive evolution: introduce a new nullable column carrying the new type going forward, keep the old column populated for historical rows, and expose a reconciled view that coalesces or casts the two so consumers see one logical column. Where the change is a safe widening, such as int to long, the format can apply it directly. Reconcile types at read time with casts or a presentation view rather than mutating files. Version the schema and communicate the deprecation, then migrate consumers gradually behind the view so none break on the switchover. Only backfill or rewrite history if a query genuinely needs the new type physically, and even then do it incrementally per partition.
The mistakes people make
Doing a destructive in-place ALTER that forces rewriting or reinterpreting every historical file at once. Breaking consumers by changing the column under them with no view or versioning. Assuming plain Parquet directories support safe evolution, when name-based schema resolution and incompatible types cause silent corruption. Rewriting petabytes when a metadata change or read-time cast would do.
What usually comes next
Which type changes are safe versus breaking. How do column IDs prevent rename issues. How do you eventually retire the old column.
A concrete example
A source changes a price field from integer cents to decimal. On an Iceberg table you add a new decimal column for new writes, keep the integer column for old rows, and publish a view that casts and coalesces both into one price column, so dashboards keep working untouched while the petabytes of history are never rewritten.
Interview question
How can you absorb a breaking column type change over petabytes without rewriting history or breaking consumers?
- a.Add a new column for the new type and expose a reconciled read-time viewCorrect
- b.Delete historical partitions older than the change
- c.Run a destructive in-place ALTER across all files
- d.Switch every consumer to read raw files directly
Why? this is the answer
Additive evolution plus a reconciling view lets new and old data coexist and keeps consumers stable without rewriting files. A destructive in-place ALTER would force a costly rewrite and risk breaking everything at once.
Just read this? Test yourself on what you have been reading.
Read the original → iceberg.apache.org
- #schema-evolution
- #iceberg
- #data-lake
- #delta-lake
- #data-engineering
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles