Handle source schema changes without downtime

What's really being asked

Whether you understand schema evolution as a versioned, mostly additive discipline rather than a risky in-place mutation, and whether you can keep producers and consumers working across the transition without losing history.

The full answer

Distinguish additive from breaking changes. Adding a new column is backward compatible: define it as nullable with a default so existing queries and old files that lack it still read correctly. The data-type change is the harder case. Avoid mutating existing data in place. Prefer a table format built for evolution, such as Apache Iceberg, Delta Lake, or Hudi, which track schema by column id and support safe type promotions like int to long. When a change is not a safe promotion, add a new typed column, backfill it from the old one, validate, then deprecate the old column. Version the schema in a registry, validate incoming records at ingest, and route nonconforming rows to a dead-letter or quarantine area instead of dropping them.

The mistakes people make

Running a destructive ALTER COLUMN that rewrites the whole table, blocking readers and risking truncation. Silently dropping records that no longer match. Assuming all consumers update at the same instant.

What usually comes next

How does column-id-based evolution differ from name-based? What is a safe versus unsafe type promotion? How do you coordinate the cutover with downstream teams?

A concrete example

A source promotes an amount field from integer to decimal and adds a currency column. The engineer adds currency as nullable, and because the table is on Iceberg, applies the int-to-decimal change as a tracked schema update without rewriting old files. New writes carry the new types, old files still read, downstream jobs are migrated one by one, and nothing is lost.