How would you use distributed tracing to debug a deployment latency issue?

WHAT THIS TESTS: This question probes whether you understand the observability signal hierarchy and when distributed tracing is the right tool for regression analysis. Metrics tell you that latency increased and logs tell you that something happened, but only traces let you follow a single causal request across service boundaries to find exactly where time was lost. The interviewer wants to see that you think in terms of request paths, not just dashboards.

A GOOD ANSWER COVERS: Four things in order. First, isolation: you narrow the investigation to the new deployment by filtering traces using deployment tags, version attributes, or timestamps aligned with the rollout. Second, localization: you identify the specific service and span where duration increased relative to a pre-deployment baseline, looking at span self-time versus child time to distinguish network latency from processing latency. Third, comparison: you explain that metrics only show aggregate percentiles and cannot isolate individual anomalous requests, while logs are uncoordinated events that lack timing context across services; traces provide the end-to-end waterfall view that connects both. Fourth, action: once the regressed span is found, you link it back to the code change, configuration diff, or downstream dependency introduced in the deployment.

COMMON WRONG ANSWERS: Treating traces as just structured logs with trace IDs is a major red flag; it misses the parent-child timing relationships that make traces valuable. Another mistake is saying you would look at the P99 metric and then grep logs for errors, which ignores the fact that latency spikes often occur without errors. Some candidates also forget to mention baseline comparison, leaving no way to prove the new deployment caused the regression. Finally, confusing span self-time with total span time can lead to blaming the wrong service.

LIKELY FOLLOW-UPS: How do you handle high-cardinality trace data in production without breaking the bank? What sampling strategy would you use if the latency issue only affects one in a thousand requests? How do you correlate traces with logs and metrics without context switching? Can you walk through a real trace waterfall and explain what each bar means?

ONE CONCRETE EXAMPLE: Imagine a checkout service deploys at 14:00 and P99 latency jumps from 200 ms to 800 ms. Metrics show the spike but not the cause. Logs show no errors. Using distributed tracing, you filter for traces tagged with version 2.3.1 and find that every slow request spends 600 ms in the payment service authorization span. Comparing to a pre-deployment trace from version 2.3.0, that same span took 50 ms. You drill into the span attributes and see a new retry policy was added with an aggressive timeout. The trace gives you the exact line of code and the downstream impact, while metrics only alerted you and logs only confirmed the service received requests.