Auto-discovering app pods for Prometheus scraping

WHAT THIS TESTS This checks whether you reach for dynamic Kubernetes service discovery rather than brittle static targets, and whether you understand relabeling.

A GOOD ANSWER COVERS The core mechanism is kubernetes_sd_configs in a scrape_config. With role pod, Prometheus queries the API server and discovers every pod as a potential target, attaching metadata labels prefixed __meta_kubernetes_pod_. You then use relabel_configs to filter and rewrite. A common convention is to keep only pods that carry the annotation prometheus.io/scrape set to true, using a keep action on the corresponding meta label. Further relabel rules read prometheus.io/path and prometheus.io/port annotations to override the metrics path and the target port, and map pod and namespace into clean labels. Because discovery is dynamic, new pods of the app are scraped automatically as they appear and removed when they die. In a managed setup using the Prometheus Operator, you skip raw config and create a PodMonitor or ServiceMonitor custom resource whose selector matches the app's labels; the Operator generates the scrape config for you.

COMMON WRONG ANSWERS Using static_configs with hardcoded pod IPs is the big mistake; pod IPs are ephemeral and change on every reschedule. Forgetting relabeling means you scrape every pod indiscriminately or none. Confusing role pod with role endpoints when you actually want per-pod scraping.

LIKELY FOLLOW-UPS Difference between role pod, endpoints, and service. How ServiceMonitor differs from PodMonitor. Why annotations versus label selectors.

ONE CONCRETE EXAMPLE You annotate the Deployment's pod template with prometheus.io/scrape true, prometheus.io/port 8080, prometheus.io/path /metrics. A scrape job with role pod and matching relabel keep rules then auto-discovers and scrapes every replica, including ones created later by scaling.