Prometheus Exemplars: Link Your Metrics to Traces
Exemplars are like footnotes for your metrics, linking a data point like a latency spike directly to a specific trace ID. This lets you jump from a 'what' on a dashboard to the 'why' in your tracing system.
Why it exists
Metrics tell you what happened (e.g., p99 latency is high), but not why. Tracing explains why a specific request was slow. Before exemplars, connecting a metric spike to the specific trace that caused it was a manual guessing game. Exemplars were created to forge a direct, machine-readable link between the symptom (metric) and the cause (trace).
The mental model
An exemplar is a footnote for a metric. When your monitoring system ingests a metric, like a request duration, it can attach a small piece of metadata—the exemplar. This metadata is not another metric; it's a reference, typically a trace_id, pointing to a different system (like Jaeger or Zipkin) that has the full story. It connects the aggregate view (metrics) to a specific, concrete example (a trace).
How it works
The OpenMetrics format allows an instrumented application to attach exemplars to metrics it exposes. In Prometheus, you must enable this feature via the --enable-feature=exemplar-storage flag. When enabled, Prometheus stores exemplars in a fixed-size circular buffer in memory for each time series. When the buffer is full, the oldest exemplar is dropped. This means you only keep the most recent examples. A typical exemplar with a trace ID uses roughly 100 bytes of memory.
When to use it
Use exemplars to drastically reduce debugging time. They are perfect for connecting a high-level dashboard to low-level diagnostics. When you see a spike in your p99 latency graph in Grafana, you can click on that spike and get a direct link to the trace that represents a request from that high-latency bucket. This shortens Mean Time to Resolution (MTTR) by eliminating guesswork.
When not to use it
Do not treat exemplars as a comprehensive logging or tracing system. They are samples, not an exhaustive record. Due to the circular buffer storage, you are only guaranteed to have recent examples. They are not suitable for auditing or getting a complete list of every event. They also add memory overhead, so be mindful of the cost on systems with very high cardinality.
One canonical example
A web service exposes a histogram metric for request latency. A slow request takes 3 seconds. The application, which is also instrumented for tracing, attaches the request's trace_id as an exemplar to that 3-second observation. Prometheus scrapes the metric and stores the value in the correct histogram bucket, along with the exemplar {trace_id="abc-123"}. Later, an engineer investigating a latency alert can retrieve this trace ID directly from the metric data in their dashboard.
Interview question
A developer wants to use Prometheus Exemplars to ensure a complete, auditable record of all slow requests over the past month. Why is this an inappropriate use case for exemplars?
- a.Exemplars only support linking to logs, not to detailed traces from systems like Jaeger or Zipkin.
- b.Exemplars introduce too much overhead for high-cardinality metrics, making them impractical for any production use.
- c.Exemplars store only a fixed-size, recent sample of events in memory, not a comprehensive historical record.Correct
- d.Exemplars are primarily designed for aggregating data, not for individual request details.
Why? this is the answer
The card states exemplars are "samples, not an exhaustive record" and are stored in a "fixed-size circular buffer in memory," meaning they only keep recent examples and are "not suitable for auditing or getting a complete list of every event." Option D is incorrect because exemplars link aggregate metrics to specific trace details. Option A is incorrect as the card explicitly mentions linking to trace IDs for systems like Jaeger or Zipkin. Option B is an exaggeration; while they add overhead, the card advises mindfulness, not universal impracticality.
Just read this? Test yourself on what you have been reading.
Read the original → prometheus.io
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on prometheus — each one lists the topics its interview covers.
See open roles