Sampling: Tracing Everything Without Storing Everything
Sampling makes high-volume observability affordable by deciding which traces to keep and which to discard. It's essential in distributed systems where capturing every request is too costly.
Why it exists
Modern distributed systems generate a massive volume of telemetry data (traces, metrics, logs) for every user request. Storing and analyzing 100% of this data is prohibitively expensive and can overwhelm observability backends, slowing down analysis. Sampling was created to solve this cost-versus-visibility problem.
The mental model
Think of sampling like quality control on a factory assembly line. You can't afford to test every single item that comes off the line. Instead, you devise a strategy to inspect a representative subset—maybe every 100th item, or any item that looks unusual—to get a clear picture of the overall production quality. Sampling applies this logic to your system's requests.
How it works
Sampling strategies determine whether a trace will be kept or discarded. There are two main approaches:
Head-Based Sampling: The decision is made at the beginning of a trace, when the first service receives a request. It's simple and efficient. For example, you can configure an OpenTelemetry SDK to keep only 1% of all traces. The major drawback is that you might discard a trace that later results in a critical error, losing valuable debugging information.
Tail-Based Sampling: The decision is made after the entire trace has completed. All spans for a trace are buffered and sent to a collector, which analyzes the complete trace. It can then decide to keep the trace based on interesting properties, like the presence of an error, high latency, or hitting a specific critical service. This is far more intelligent but requires more infrastructure to buffer and process traces before making a decision.
When to use it
Sampling is a necessity for almost any high-traffic production system, especially those with microservice or serverless architectures. If a single user action can generate dozens or hundreds of spans across services, sampling is the only way to manage observability costs and maintain backend performance. It turns an unmanageable firehose of data into a useful, representative dataset.
When not to use it
In very low-traffic environments, like a personal project or a development server, you might sample 100% of traces (i.e., disable sampling) to see everything. Similarly, for specific, critical, low-volume business transactions (like a final payment confirmation), you might configure rules to always keep those traces for audit purposes, while sampling everything else.
One canonical example
A checkout service processes 1,000 requests per second. With head-based sampling at 1%, the system decides upfront to keep 10 traces per second, dropping the other 990. If a rare database deadlock causes an error on one of the dropped traces, the evidence is lost. With tail-based sampling, all 1,000 traces/sec are briefly analyzed. The sampler sees the trace with the database error and high latency and flags it to be saved, while discarding the 999 successful traces. This guarantees you capture the interesting failures.
Interview question
In a high-traffic distributed system, which sampling strategy is most effective for ensuring that traces containing critical errors or high latency are reliably captured, despite increased infrastructure overhead?
- a.Tail-based sampling, as it analyzes the complete trace before deciding to keep it.Correct
- b.Head-based sampling, because it makes decisions early, reducing processing load.
- c.Implementing a fixed-rate sampling approach at the service entry point.
- d.Disabling sampling entirely, to guarantee 100% visibility of all events.
Why? this is the answer
Tail-based sampling evaluates the entire trace, including its outcome (errors, latency), before deciding to keep it, thus guaranteeing capture of interesting failures. Head-based sampling decides upfront and might discard traces that later become critical, making it unsuitable for reliably capturing all errors in high-traffic systems.
Just read this? Test yourself on what you have been reading.
Read the original → opentelemetry.io
- #observability
- #sampling
- #distributed tracing
- #sre
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on observability — each one lists the topics its interview covers.
See open roles