How do you manage an event schema change?
This tests your understanding of data contracts and backward compatibility. A great answer details a phased, backward-compatible rollout (add nullable field), starting with consumers (warehouse) then producers, and mentions using a schema registry for…
What's really being asked
This question tests your ability to think about a data system holistically, from producer to consumer. Interviewers want to see that you understand schemas as contracts, the importance of backward compatibility, and the operational steps required to roll out a change without breaking existing systems or losing data. It's a test of operational maturity and systems thinking, not just a SQL quiz.
The full answer
A strong answer outlines a phased, backward-compatible approach. First, define the change as adding a new, nullable field ('attribution_source'), which is a backward-compatible change that won't break older consumers. Second, detail the rollout order: update downstream consumers (the data warehouse) to handle the new optional field first. This typically involves an ALTER TABLE statement to add the nullable column. Third, update the data producers (the application code) to start sending the new field. Fourth, mention using a schema registry (like Confluent Schema Registry for Kafka) to enforce and version these contracts programmatically, ensuring producers and consumers agree on the schema.
The mistakes people make
A common red flag is suggesting a breaking change without a clear, multi-stage migration plan, like making the new field mandatory immediately. Another is focusing solely on the data warehouse (ALTER TABLE my_table ADD COLUMN...) while ignoring the upstream producers and data transport layer; this shows a siloed, non-systems perspective. A junior answer might also suggest dropping and recreating the table, which is unacceptable due to data loss and downtime. These approaches lead to the exact pipeline failures and silent data loss the interviewer is looking for you to avoid.
What usually comes next
"How would you handle a breaking change, like renaming or deleting a field?" (Requires a multi-step process: add new field, backfill data, dual-write, migrate all consumers, then drop old field). "What if the data warehouse table is 50TB? Does that change your ALTER TABLE approach?" (Yes, it may require online schema change tools or platform-specific features to avoid locking the table for hours or days). "How do you communicate this change to analysts and other data consumers?" (Through versioned documentation, release notes, and announcements in a shared channel).
A concrete example
To add attribution_source to a signup event in a Kafka pipeline feeding a Snowflake warehouse: 1. In Snowflake, run ALTER TABLE signups ADD COLUMN attribution_source VARCHAR NULL;. This is a metadata-only, non-blocking operation that prepares the destination. 2. In your Schema Registry, register a new version of the signup Avro schema that includes the new optional field with a default null value. 3. Deploy the new producer code that sends the attribution_source field. Old producers can still send the old schema version without issue. 4. After deployment, verify data is flowing correctly. Historical data will have NULL for this column, which is expected and doesn't break existing reports that don't use the new field.
Interview question
What is the recommended sequence for introducing a new, nullable field to an existing event schema in a production data pipeline?
- a.Update producers to send the new field, then update the schema registry, and finally update consumers to accept it.
- b.Update downstream consumers to accept the new nullable field, then register the updated schema, and finally deploy producers to send the new field.Correct
- c.Add the new column to the data warehouse, then notify producers to start sending the data, bypassing a schema registry for nullable fields.
- d.Register the new schema with the nullable field, then update all producers and consumers simultaneously to use the new schema.
Why? this is the answer
The correct approach is to first prepare downstream consumers to handle the new nullable field, then update the schema in the schema registry, and finally deploy producers. This phased rollout ensures backward compatibility and prevents data loss or errors. Option A is incorrect because updating producers before consumers can break existing pipelines.
Just read this? Test yourself on what you have been reading.
Read the original → branchboston.com
- #data engineering
- #schema evolution
- #system design
- #analytics
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on data engineering — each one lists the topics its interview covers.
See open roles