Explain data lineage and how you would implement it

WHAT THIS TESTS: This question assesses your practical system design skills for data observability and governance, not just a textbook definition. The interviewer wants to see if you can architect a solution for a complex, real-world problem: tracking data flow in a distributed microservices environment to enable faster debugging and impact analysis. They are testing your knowledge of modern data tooling and the trade-offs involved.

A GOOD ANSWER COVERS: A strong answer has three parts. First, define data lineage concisely: it's the process of tracking data's lifecycle, documenting its origin, transformations, and movement to provide visibility for root cause analysis. Second, propose a concrete implementation strategy. This involves instrumentation at the source (microservices) and during transformation (ETL/ELT jobs). Mention using an open standard like OpenLineage to emit lineage events. Third, describe the backend system. These events are sent to a central service like Marquez or DataHub, which collects the metadata, stores it in a graph database, and provides a UI to visualize the entire data flow from a service's database table to a dashboard.

COMMON WRONG ANSWERS: A major red flag is giving only a high-level definition without any implementation details. Another weak answer is suggesting a manual approach, like documenting lineage in a wiki or spreadsheet. This is completely unscalable for a microservices architecture with dozens or hundreds of services and signals inexperience. Failing to mention the 'why' (root cause analysis, impact analysis) shows a lack of business context. Simply listing tools without explaining how they fit together is also a weak answer.

LIKELY FOLLOW-UPS: Expect questions about trade-offs. How much performance overhead does this add to your services? How do you handle schema changes and evolution? What's the difference between table-level lineage and column-level lineage, and when would you need the latter? How would you bootstrap this system for an existing, large-scale environment with hundreds of pipelines?

ONE CONCRETE EXAMPLE: Imagine a 'user_signups' table owned by the AuthService. An ETL job reads this table, joins it with data from a BillingService, and writes to an 'active_users' table in the data warehouse. A dashboard then queries this table. If the dashboard numbers look wrong, lineage allows an analyst to immediately see all upstream dependencies. They can trace the data back from the dashboard, through the 'active_users' table, to the ETL job, and finally to the source tables in the AuthService and BillingService, reducing root cause analysis from hours to minutes.