Prometheus
25 bites tagged Prometheus — interview questions with model answers, and 60-second explainers.
Alert silencing versus alert inhibition
Silencing mutes by matcher for a window, inhibition suppresses lower alerts when a higher one fires, dependency-aware. Alertmanager noise control. treating them as synonyms or over-silencing real signals.
Writing SLIs in PromQL
Availability is good requests over total using rate and non-5xx counters; latency uses histogram_quantile over rate of buckets summed by le. turning SLOs into correct PromQL.
Scaling Prometheus for HA and volume
Shard scraping, add long-term object storage and global query via Thanos/Cortex/VictoriaMetrics, run redundant replicas for HA, with dedupe. scaling a single Prometheus past its limits.
Prometheus histogram versus summary
Histograms expose bucket counts and let you compute aggregatable quantiles at query time; summaries compute fixed quantiles per instance that cannot be combined. where quantiles are computed and whether they aggregate.
High cardinality in time-series databases
Each combination is a separate series consuming memory and index; manage by avoiding unbounded labels and bucketing values. knowing cardinality equals number of unique label combinations.
Core components of Prometheus
Server scrapes targets found via service discovery, stores samples in a local time-series database, with Alertmanager and exporters as helpers. understanding Prometheus's pull model and TSDB.
Histograms versus summaries for latency
Histograms store bucket counts and stay aggregatable with query-time approximate quantiles; summaries precompute quantiles per instance that cannot be averaged. understanding where quantiles are computed and whether they aggregate.
Redesigning a high-cardinality request metric
Cardinality is unique label combinations; user_id and raw path are unbounded; redesign by dropping user_id and templating the path. Diagnosing and fixing label explosion.
What is high-cardinality data in Prometheus?
Cardinality is the count of unique label combinations; each is a separate series; explosion blows up memory and query cost. Understanding time-series storage cost.
Metrics in Observability
Metrics are numeric measurements aggregated over time, like counters, gauges, and histograms. They are cheap to store and fast to query, making them ideal for dashboards and alerting, but their pre-aggregation discards per-event detail needed for deep…
Alerting on under-replicated Deployments
Write an alerting rule comparing kube_state_metrics available vs desired replicas with for: 5m, Prometheus evaluates and fires to Alertmanager, which dedupes/routes/notifies. Prometheus alerting pipeline.
PromQL for top 5 CPU-consuming pods
Apply rate() to the counter over 15m, sum by pod to combine containers, then wrap in topk(5); rate handles counter resets. PromQL on counters.
Auto-discovering app pods for Prometheus scraping
Use kubernetes_sd_configs with role pod, relabel on pod annotations like prometheus.io/scrape to filter, and set path and port; with the Operator use a PodMonitor or ServiceMonitor. Kubernetes service discovery in Prometheus.
kube-state-metrics versus node-exporter
Kube-state-metrics exposes API object state (deployment replicas, pod phase, restarts) from the control plane, while node-exporter exposes OS-level hardware metrics (CPU, memory, disk) per… Distinguishing object-state from host metrics.
Multi-tenant isolation with a monitoring exception
Apply default-deny ingress per tenant namespace, allow same-namespace traffic, then add an ingress rule permitting the monitoring namespace via namespaceSelector on the metrics port. Layered NetworkPolicy design.
Diagnose a Prometheus cardinality explosion
Find offenders via TSDB stats and topk count by __name__, identify unbounded labels, then drop or aggregate them with relabeling. operating Prometheus at scale. just scaling memory without fixing label design.
How would you instrument and query P95 API latency by region?
This tests white-box latency instrumentation and safe cardinality for percentile aggregation. Strong answer: emit histograms by region, query P95 with histogram_quantile or a log percentile, and keep trace IDs in logs only.
How would you measure P95 latency by geographic region?
Tests your ability to translate a business need into a concrete observability implementation. A good answer involves instrumenting the API with a histogram metric, adding a region label via GeoIP, and querying with `histogram_quantile`.
How would you measure P95 latency by geographic region?
Tests your ability to design a practical metrics pipeline, considering instrumentation, data types (metrics vs. logs), and aggregation. Instrument the API with a histogram metric and a `region` label, then query using `histogram_quantile`.
Query Federation: Combining Prometheus Servers
Query federation lets one Prometheus server scrape metrics from another, creating a meta-monitor. Use it to build a global view from local servers or to combine application and infrastructure metrics for richer alerts.
Prometheus Exemplars: Link Your Metrics to Traces
Exemplars are like footnotes for your metrics, linking a data point like a latency spike directly to a specific trace ID. This lets you jump from a 'what' on a dashboard to the 'why' in your tracing system.
Cardinality: The Hidden Cost of Time-Series Metrics
Cardinality is the number of unique label combinations in your metrics. High cardinality, from labels like user IDs, is the silent killer of monitoring systems like Prometheus, exploding memory and cost. The footgun is adding a label with unbounded values.
Prometheus Alertmanager: Taming Your Alert Storms
Alertmanager is the traffic controller for your Prometheus alerts, turning a potential flood into actionable notifications. It groups, deduplicates, and routes alerts to services like PagerDuty. The footgun: don't load balance traffic to an HA cluster.
PromQL: Querying Time Series Data as Vectors
PromQL treats metrics as vectors of values over time, letting you slice and aggregate system state. It's used for Grafana dashboards and Alertmanager rules. The footgun: applying `rate()` to a gauge instead of a counter produces silent, nonsensical results.
Get Prometheus bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.