How do you manage event schema evolution without breaking reports?

WHAT THIS TESTS: This question probes whether you see schema evolution as a distributed systems contract problem rather than a simple database migration. The interviewer cares if you understand that an analytics pipeline has multiple consumers including stream processors, batch ETL, data scientists, and BI tools, each with different tolerance for change. They want to know if you can prevent downstream breakage while enabling new analytics.

A GOOD ANSWER COVERS: First, enforce backward-compatible serialization using Avro, Protobuf, or JSON Schema with default values so old readers can deserialize new events without failing. Second, add the new attribution_source field as nullable or with a sensible default, never as a required field on existing events, which prevents pipeline crashes during the transition. Third, separate raw ingestion from modeled consumption by landing events in a raw or bronze layer that accepts schema drift, then propagating the new field through curated silver and gold tables only after validation so historical reports remain stable. Fourth, use a schema registry to version the event contract, enforce compatibility rules, and give downstream teams a changelog. Fifth, communicate the change before deployment and run a shadow or dual-write period to measure impact on dashboards and ML features.

COMMON WRONG ANSWERS: A major red flag is suggesting an ALTER TABLE DROP COLUMN or making the new field non-nullable on the raw event store, which immediately breaks old readers. Another is proposing to rewrite or delete historical data so every past signup row matches the new schema, destroying the immutable event log. Saying you will just backfill everything over a weekend without a compatibility plan also signals inexperience with production analytics uptime.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle a type change on an existing field, such as changing a timestamp from epoch seconds to an ISO string, or how you would manage a breaking change when a partner removes a field you depend on. They might also probe cost by asking how to reprocess six months of Parquet in S3 without doubling your compute bill, or how you enforce schema contracts in a loosely governed microservices environment.

ONE CONCRETE EXAMPLE: Suppose your signup events land in S3 as Parquet via Kinesis Firehose. You register version 1 of the schema in the Glue Schema Registry with backward compatibility enforced. You update the producer to emit version 2 that includes attribution_source as an optional string with a default of null. Firehose continues writing Parquet without errors because the new field is optional. In Spark, the raw table picks up the new column automatically. Your dbt model for signups selects all historical columns plus the new attribution_source, coalescing nulls to unknown for reporting. BI dashboards using the modeled table see no interruption, while analysts can immediately query the new field. After a week of validation, you mark version 2 as the default and deprecate version 1.

Source: branchboston.com