Explain event schemas and the purpose of a schema registry

WHAT THIS TESTS: This question probes your practical experience with data engineering and event-driven systems. The interviewer wants to see if you understand that event data is a public API between services and teams. They are testing your ability to articulate the need for data contracts (schemas) and enforcement mechanisms (registries) to prevent system-wide chaos, data corruption, and loss of trust in analytics as an organization grows.

A GOOD ANSWER COVERS: First, define an event schema as a formal contract or blueprint for an event's structure, specifying field names, data types (string, integer), and nullability. It's an API contract for data.

Second, describe a schema registry as a centralized, version-controlled repository for these schemas. Its primary job is to enforce compatibility rules (e.g., backward, forward compatibility) when schemas evolve. This prevents producers from shipping breaking changes that downstream consumers cannot parse.

Third, detail the specific problems that arise at scale without one. This includes brittle pipelines, where a simple field name change (user_id to userId) or data type change ("123" to 123) breaks downstream consumers, causing outages. It also includes silent data corruption, where new enum values are ignored, leading to incomplete analytics. This ultimately erodes business trust in data and wastes engineering time on debugging instead of building.

COMMON WRONG ANSWERS: Describing a schema as just "the structure of the data" without its role as a contract. Describing a registry as just a "place to store schemas" without mentioning its critical role in versioning and compatibility enforcement. Giving vague problems like "it causes data quality issues" instead of concrete examples, like a data type change causing a deserialization error that halts a Spark job.

LIKELY FOLLOW-UPS: "Describe the difference between backward and forward compatibility. When would you choose one over the other?" or "How would you roll out a schema registry in an organization that doesn't currently have one?" or "What are some popular schema formats like Avro or Protobuf, and what are their trade-offs?"

ONE CONCRETE EXAMPLE: Imagine an order_created event with a price field as a float (e.g., 19.99). A producer team changes it to an integer representing cents (1999) to avoid floating-point errors. Without a schema registry, this change is deployed. The downstream billing job, expecting a float, either crashes or misinterprets 1999 as 1999.00 instead of 19.99, causing massive reporting errors. A schema registry with backward compatibility checks would have blocked the producer's deployment, flagging the type change as a breaking change and preventing the incident.