Sampling: Tracing Everything Without Storing Everything

WHY IT EXISTS Modern distributed systems generate a massive volume of telemetry data (traces, metrics, logs) for every user request. Storing and analyzing 100% of this data is prohibitively expensive and can overwhelm observability backends, slowing down analysis. Sampling was created to solve this cost-versus-visibility problem.

THE MENTAL MODEL Think of sampling like quality control on a factory assembly line. You can't afford to test every single item that comes off the line. Instead, you devise a strategy to inspect a representative subset—maybe every 100th item, or any item that looks unusual—to get a clear picture of the overall production quality. Sampling applies this logic to your system's requests.

HOW IT WORKS Sampling strategies determine whether a trace will be kept or discarded. There are two main approaches:

Head-Based Sampling: The decision is made at the beginning of a trace, when the first service receives a request. It's simple and efficient. For example, you can configure an OpenTelemetry SDK to keep only 1% of all traces. The major drawback is that you might discard a trace that later results in a critical error, losing valuable debugging information.

Tail-Based Sampling: The decision is made after the entire trace has completed. All spans for a trace are buffered and sent to a collector, which analyzes the complete trace. It can then decide to keep the trace based on interesting properties, like the presence of an error, high latency, or hitting a specific critical service. This is far more intelligent but requires more infrastructure to buffer and process traces before making a decision.

WHEN TO USE IT Sampling is a necessity for almost any high-traffic production system, especially those with microservice or serverless architectures. If a single user action can generate dozens or hundreds of spans across services, sampling is the only way to manage observability costs and maintain backend performance. It turns an unmanageable firehose of data into a useful, representative dataset.

WHEN NOT TO USE IT In very low-traffic environments, like a personal project or a development server, you might sample 100% of traces (i.e., disable sampling) to see everything. Similarly, for specific, critical, low-volume business transactions (like a final payment confirmation), you might configure rules to always keep those traces for audit purposes, while sampling everything else.

ONE CANONICAL EXAMPLE A checkout service processes 1,000 requests per second. With head-based sampling at 1%, the system decides upfront to keep 10 traces per second, dropping the other 990. If a rare database deadlock causes an error on one of the dropped traces, the evidence is lost. With tail-based sampling, all 1,000 traces/sec are briefly analyzed. The sampler sees the trace with the database error and high latency and flags it to be saved, while discarding the 999 successful traces. This guarantees you capture the interesting failures.