How would you implement data lineage for microservices analytics?

WHAT THIS TESTS: This question evaluates whether you understand data lineage as a living graph of provenance rather than a static inventory. At the senior level, interviewers want to see systems thinking about how to capture lineage across service boundaries without adding crippling latency, how to version schemas as microservices evolve, and how to make the lineage queryable for both incident response and data governance audits.

A GOOD ANSWER COVERS: Four layers in order. First, capture: instrument microservices to emit metadata at the point of data creation or handoff, typically via OpenLineage emitters, Kafka headers, or API gateway middleware that records schema versions and dataset identifiers. Second, storage: centralize metadata in a graph-oriented catalog such as DataHub, OpenMetadata, or a custom graph database like Neo4j, storing entities as datasets, jobs, and columns with directed edges for upstream and downstream relationships. Third, granularity: insist on column-level lineage, not just table-level, because analytics pipelines often involve complex SQL transforms, joins, and aggregations where a single bad column propagates downstream; tools like dbt exposures or Spark SQL parsers can automate this. Fourth, consumption: expose lineage through a UI for discovery and via an API for programmatic impact analysis, so that when a microservice changes a schema, you can automatically notify owners of affected dashboards or models.

COMMON WRONG ANSWERS: Proposing manual spreadsheets or Confluence tables that inevitably rot. Describing lineage as just logging or observability without a queryable graph structure. Suggesting that the data warehouse alone is sufficient, which ignores the critical upstream microservice boundary where data is born. Recommending heavy synchronous instrumentation that adds latency to production API calls. Claiming that lineage is only useful for compliance rather than operational debugging.

LIKELY FOLLOW-UPS: How would you handle schema evolution in upstream services without breaking lineage links? What is your strategy for column-level lineage when data passes through Python or Spark jobs with dynamic logic? How do you balance completeness versus the performance cost of metadata collection in high-throughput services? Would you store lineage metadata in the same datastore as the analytics or keep it separate?

ONE CONCRETE EXAMPLE: Imagine an e-commerce pipeline. An Order Service emits events to Kafka. An analytics job reads those events, joins with a User Service snapshot in S3, and writes a fact table to Snowflake. A good implementation uses OpenLineage integrated with the Kafka producer to emit dataset and run metadata, a Spark OpenLineage listener to capture the join logic and column mappings, and DataHub as the central catalog. When the Order Service deprecates a field, the catalog flags the three downstream Looker dashboards and the dbt model that depend on it, letting you trace the impact in seconds rather than hours and schedule coordinated migrations.