Schema evolution without rewriting history

WHAT THIS TESTS This evaluates whether you can manage breaking schema changes over petabyte-scale history using metadata-aware techniques instead of brute-force rewrites that are slow, expensive, and risky.

A GOOD ANSWER COVERS The goal is to absorb the source's change while leaving the existing data files untouched and downstream consumers working. Modern table formats are the key enabler: Apache Iceberg and Delta Lake track schema using stable column identifiers and a metadata layer, so operations like adding a column, renaming, or widening a compatible type are metadata-only and do not rewrite data. For a genuinely incompatible type change, prefer additive evolution: introduce a new nullable column carrying the new type going forward, keep the old column populated for historical rows, and expose a reconciled view that coalesces or casts the two so consumers see one logical column. Where the change is a safe widening, such as int to long, the format can apply it directly. Reconcile types at read time with casts or a presentation view rather than mutating files. Version the schema and communicate the deprecation, then migrate consumers gradually behind the view so none break on the switchover. Only backfill or rewrite history if a query genuinely needs the new type physically, and even then do it incrementally per partition.

COMMON WRONG ANSWERS Doing a destructive in-place ALTER that forces rewriting or reinterpreting every historical file at once. Breaking consumers by changing the column under them with no view or versioning. Assuming plain Parquet directories support safe evolution, when name-based schema resolution and incompatible types cause silent corruption. Rewriting petabytes when a metadata change or read-time cast would do.

LIKELY FOLLOW-UPS Which type changes are safe versus breaking. How do column IDs prevent rename issues. How do you eventually retire the old column.

ONE CONCRETE EXAMPLE A source changes a price field from integer cents to decimal. On an Iceberg table you add a new decimal column for new writes, keep the integer column for old rows, and publish a view that casts and coalesces both into one price column, so dashboards keep working untouched while the petabytes of history are never rewritten.