Distributed tracing for serverless apps

WHAT THIS TESTS This evaluates whether you can build the three pillars of observability, traces, metrics, and logs, into a distributed serverless system so a single slow request is diagnosable.

A GOOD ANSWER COVERS Distributed tracing hinges on context propagation. A trace ID is created at the entry point, typically the API gateway, and passed via headers through every function and downstream call so all spans roll up into one trace. Instrument each component with OpenTelemetry or a provider tracer such as X-Ray, creating spans for the gateway, each function invocation, and each external call including database queries, so you see exactly which hop consumes the latency. Emit structured JSON logs that include the trace ID and a request correlation ID so logs join to traces. Capture metrics such as per-function duration, error rate, throttles, and cold-start counts. Send everything to a backend where you can view the waterfall, spot the bottleneck span, and drill into the correlated logs. Sample intelligently to control cost while always capturing errors.

COMMON WRONG ANSWERS Relying on separate, uncorrelated logs per function so you cannot reconstruct a request path. Forgetting to instrument the database call, often the actual bottleneck. Failing to propagate trace context across async boundaries such as queues. Logging unstructured text that is hard to query. Sampling so aggressively that error traces are lost.

LIKELY FOLLOW-UPS How do you propagate context through an async queue? What sampling strategy balances cost and coverage? How do spans and logs correlate? How much latency does instrumentation add?

ONE CONCRETE EXAMPLE A request is slow. The trace waterfall shows the gateway and two functions are fast, but a 400ms span on a single database query dominates the request. The correlated log line for that span reveals a missing index on the queried column; adding the index collapses that span and the end-to-end latency drops sharply.