Query Federation: Combining Prometheus Servers
Query federation lets one Prometheus server scrape metrics from another, creating a meta-monitor. Use it to build a global view from local servers or to combine application and infrastructure metrics for richer alerts.
Why it exists
As monitoring setups grow, a single Prometheus server can become a bottleneck. Furthermore, metrics for a single service might be split across different logical systems (e.g., app metrics vs. cluster resource metrics). Query federation was created to solve both problems by allowing Prometheus servers to selectively share data with each other.
The mental model
Think of federation as a tree or a mesh of Prometheus servers, not a single monolith. Instead of one giant server trying to scrape everything, you have specialized servers that can query each other. A global server might ask its regional children, "Give me your aggregated API error rates," but not the noisy, instance-level details.
How it works
A source Prometheus server exposes a special /federate HTTP endpoint. A second, scraping Prometheus server is configured to hit this endpoint. The request must include at least one match[] URL parameter, which specifies an instant vector selector (like {job="api-server"}) to choose which time series to retrieve. The source server then returns the current value for all matching series.
When to use it
Two primary scenarios. First, Hierarchical Federation: Use this to scale. Local Prometheus servers in each datacenter collect high-detail metrics. A central, global Prometheus server then federates aggregated data (e.g., job-level summaries) from these local servers for a high-level overview. Second, Cross-Service Federation: Use this to combine related datasets. An application's Prometheus can pull CPU and memory metrics for its specific instances from a cluster-wide Prometheus, enabling alerts that correlate app behavior with resource consumption.
When not to use it
Federation is not a backup or long-term storage solution. Do not use it to copy all time series from one server to another; this is inefficient and defeats the purpose. For durable, long-term storage of all metrics, use Prometheus's remote write feature to send data to a specialized time-series database. Federation is for live, selective querying between active servers.
One canonical example
A company runs services across three datacenters. Each has its own Prometheus server scraping local metrics. A single "global" Prometheus server is configured to federate data from these three servers. It doesn't pull every metric. Instead, it uses match[] to pull only high-level, aggregated metrics like job:http_requests:rate5m. This gives the global SRE team a worldwide view of service health without storing terabytes of redundant, instance-level data.
Interview question
A company needs to consolidate all time series data from multiple regional Prometheus servers into a central system for durable, long-term storage. Which Prometheus feature is most appropriate?
- a.Implementing Hierarchical Federation, where the central server pulls aggregated, high-level metrics from the regional servers for a global overview.
- b.Employing Cross-Service Federation to combine application and infrastructure metrics from different regional servers.
- c.Utilizing Prometheus's remote write feature to send all collected metrics from each regional server to a dedicated time-series database.Correct
- d.Configuring the central Prometheus server to use Query Federation to scrape all metrics from each regional server's /federate endpoint.
Why? this is the answer
The card explicitly states that federation is not for copying all time series or for long-term storage. For durable, long-term storage of all metrics, Prometheus's remote write feature to a specialized time-series database is the recommended solution.
Just read this? Test yourself on what you have been reading.
Read the original → prometheus.io
- #prometheus
- #monitoring
- #sre
- #federation
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on prometheus — each one lists the topics its interview covers.
See open roles