Explain data lineage and how you'd implement it

WHAT THIS TESTS: This question tests your understanding of data governance principles and your ability to architect a practical solution in a modern, distributed environment (microservices). The interviewer is looking for more than a textbook definition. They want to see if you can reason about metadata collection, storage, and visualization, and understand the tradeoffs involved. It's a test of both conceptual knowledge and systems design skills applied to data.

A GOOD ANSWER COVERS: A strong answer has three parts. First, define data lineage clearly: it's the documented lifecycle of data, tracking its origin, what transformations are applied, and where it moves. Second, propose a concrete implementation strategy for a microservices environment. This involves instrumenting services to emit lineage metadata (e.g., "Service A read table X and wrote to topic Y"), often using a standard like OpenLineage. Third, describe the architecture for processing and visualizing this metadata. This includes a central metadata store (a graph database like Neo4j is ideal for modeling relationships) and a discovery/visualization tool (like Amundsen, Marquez, or DataHub) that queries this store to show the end-to-end flow.

COMMON WRONG ANSWERS: A major red flag is providing only a vague, academic definition of data lineage without any practical implementation details. Another common mistake is naming a tool ("We'd use Amundsen") without explaining the underlying architecture required to feed it data. Simply saying "we'll log everything" is also a poor answer; it misses the need for structured, event-based metadata that describes the relationships between data assets, not just raw log files. Candidates also fail when they can't explain why lineage is important, such as for root cause analysis of bad data, impact analysis for schema changes, or regulatory compliance.

LIKELY FOLLOW-UPS: Expect questions about scaling the metadata collection system. How would you handle millions of events per day? How would you ensure the lineage data itself is accurate and not stale? Another follow-up could be about "column-level lineage": how would you track that a specific column in a downstream table was derived from two specific columns in upstream sources? This tests deeper knowledge of the complexity involved.

ONE CONCRETE EXAMPLE: For an analytics pipeline, imagine a user_signup event from a UserService is published to a Kafka topic. A FraudDetectionService consumes this, enriches it with risk data, and writes a user_fraud_score record to a database table. An AnalyticsETL job then reads this table, joins it with ProductUsage data, and writes the final aggregated data to a DailyMetrics table in the data warehouse. Data lineage would visually map this entire flow: UserService -> Kafka topic -> FraudDetectionService -> DB table -> AnalyticsETL -> Data Warehouse table. This allows an analyst to see exactly where the DailyMetrics data came from.