Diagnosing latency with distributed tracing

What's really being asked

Whether you can move from a vague latency complaint to the exact slow hop using traces, and whether you know the one thing the mesh cannot do alone: propagate context.

The full answer

The mesh sidecars automatically generate a span for each inbound and outbound hop and attach timing, status and identity. To stitch those spans into a single end-to-end trace, the application must forward the tracing headers, such as the B3 or W3C traceparent headers, from inbound to outbound requests; the mesh injects them but cannot carry them across an app's internal call boundary. With that in place, you capture a representative slow request, open its trace in the tracing UI, and read the waterfall. You look for the span with the largest self time, distinguishing time spent inside a service from time waiting on a downstream dependency, and from network or queueing delay visible as gaps between spans. Aggregate per-service latency dashboards point you at suspects, but the individual trace localizes the bottleneck.

The mistakes people make

Claiming the mesh produces complete traces with zero application involvement; without header propagation traces fragment into disconnected single-hop spans. Relying only on average or p99 dashboards, which show that something is slow but not where. Confusing logs and metrics with tracing; each answers a different question.

What usually comes next

Why does sampling matter and how do you ensure the slow request was sampled? What headers must the app propagate and how? How do you separate genuine service compute time from lock contention or GC pauses that a trace span alone may not explain? How do metrics and traces complement each other?

A concrete example

The checkout call spans gateway, cart, pricing and inventory. The trace waterfall shows gateway, cart and inventory each finishing in tens of milliseconds, but the pricing span has 800 milliseconds of self time with no child calls, pointing at slow computation inside pricing rather than a network or downstream issue. You confirm by correlating that trace's timestamp with pricing's CPU and GC metrics.