How do you root-cause bad data across microservices and Spark?

WHAT THIS TESTS: This question evaluates whether you can methodically debug a distributed data pipeline without guessing. Interviewers want to see if you understand data lineage, schema evolution, late-arriving data, and idempotency in Spark. They also care if you can design observability proactively rather than treating debugging as a purely reactive firefight.

A GOOD ANSWER COVERS: First, contain the blast radius by identifying which report cells are wrong and when the anomaly started. Second, trace data lineage backward from the mart through each Spark stage to the microservice sources, checking row counts, checksums, and freshness timestamps at every handoff. Third, validate schema contracts between producers and consumers because silent schema drift is a common culprit. Fourth, examine Spark internals like partition skew, watermarks, and exactly-once semantics to rule out transformation bugs. Fifth, describe the tooling you wish existed such as column-level lineage systems like OpenLineage, automated data diff tools like Datafold or dbt tests, immutable audit logs, and freshness alerts on every microservice API boundary.

COMMON WRONG ANSWERS: A major red flag is suggesting a full reprocess of the pipeline without first isolating the bad stage. Another is blaming the data mart without checking upstream freshness or microservice payload changes. Candidates who ignore schema drift or assume all microservices use the same timezone and cutoff windows also signal inexperience. Saying you would manually query every table without a systematic bisection strategy suggests you have not operated at scale.

LIKELY FOLLOW-UPS: The interviewer might ask how you would prevent this from recurring, which should lead to data contracts, schema registries, and automated quality gates like Great Expectations. They might also ask how you handle late-arriving data in Spark streaming, or how you reconcile exactly-once semantics with idempotent loads into the data mart. Another angle is cost versus thoroughness and how you balance deep lineage tracing with the urgency of a financial close.

ONE CONCRETE EXAMPLE: Suppose a revenue report is off by two percent for the last three days only. You check the mart and see correct totals for earlier periods. You trace lineage to the Spark aggregation job and notice the row count dropped on a specific run. You diff the microservice API payloads for that date range and discover a new field was added silently, causing a JSON parser to drop records in the ingestion stage. You fix the schema registry entry, backfill the affected partition, and add a Great Expectations row count check plus an OpenLineage hook to catch schema changes before they hit Spark.