Prometheus Architecture: A Pull-Based Monitoring System
Prometheus is a monitoring system that actively pulls metrics from your services, rather than waiting for them to push data. It's the standard for tracking performance in dynamic environments like Kubernetes.
Why it exists
Modern systems, especially in containerized environments like Kubernetes, are dynamic. Services are ephemeral, scaling up and down constantly. A monitoring system needs to handle this constant change, discovering what to monitor and collecting data reliably without requiring complex logic on every application.
The mental model
Prometheus operates on a "pull" model. Imagine it as a census taker that regularly visits each house (your service endpoint) on its route to ask for its status (collect metrics). It doesn't wait for services to mail in their data. This centralizes the collection logic and keeps the monitored services simple; they just need to expose an HTTP endpoint with their current state.
How it works
The core is the Prometheus server, which scrapes and stores time-series data. It finds targets to monitor using static configuration or dynamic service discovery (e.g., by querying the Kubernetes API). At a regular interval, the server sends an HTTP GET request to each target's /metrics endpoint. The target responds with its current metrics in a simple text format. The server ingests this data, stores it locally, and makes it available for querying via PromQL. For alerting, the server evaluates rules and forwards alerts to a separate Alertmanager component.
When to use it
Use Prometheus for monitoring the health and performance of long-running services like web servers, APIs, and databases. It excels in dynamic environments like Kubernetes where its service discovery automatically tracks new and removed pods. It is the go-to for numerical, time-series data like request counts, latencies, error rates, and resource usage.
When not to use it
Prometheus is not for event logging or distributed tracing; it's for aggregated metrics, not individual events. Its pull model is a poor fit for short-lived batch jobs or serverless functions that may not exist when the scrape occurs. For these, an intermediary Pushgateway is needed, which adds complexity and becomes a single point of failure.
One canonical example
A Prometheus server runs inside a Kubernetes cluster. It uses the Kubernetes API for service discovery to find all pods with the annotation prometheus.io/scrape: 'true'. Every 30 seconds, it scrapes the /metrics endpoint on those pods to collect application-specific metrics. A separate Grafana instance then queries Prometheus to build dashboards showing cluster-wide CPU usage, memory consumption, and API request latency.
Interview question
What is a primary advantage of Prometheus's pull-based metric collection in dynamic environments?
- a.It centralizes the responsibility for metric collection and target discovery within the Prometheus server.Correct
- b.It eliminates the need for services to expose a dedicated metrics endpoint, simplifying application development.
- c.It allows services to push metrics only when significant changes occur, optimizing network usage.
- d.It ensures that metrics are always collected in real-time, providing immediate insights into system state.
Why? this is the answer
Prometheus's pull model centralizes collection logic and service discovery, allowing the server to find and scrape metrics from dynamic targets, simplifying the monitored services. Option C describes a push model, which is the opposite of Prometheus's core design.
Just read this? Test yourself on what you have been reading.
Read the original → prometheus.io
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on monitoring — each one lists the topics its interview covers.
See open roles