Alerting on under-replicated Deployments

WHAT THIS TESTS This checks that you can both write a sensible alerting expression and articulate the full alerting pipeline, including the often-misunderstood role of the for clause and Alertmanager.

A GOOD ANSWER COVERS The data comes from kube-state-metrics, which exposes kube_deployment_spec_replicas, the desired count, and kube_deployment_status_replicas_available, the currently available count. You write an alerting rule in Prometheus whose expr fires when available is below desired, for example kube_deployment_status_replicas_available < kube_deployment_spec_replicas, joined on the deployment and namespace labels. Crucially you add for: 5m, meaning the condition must stay true for five continuous minutes before the alert transitions from pending to firing; this suppresses transient dips during normal rolling updates. You attach labels like severity and annotations with a human summary. The pipeline: Prometheus evaluates rules on its evaluation interval, marks matching series pending then firing once for elapses, and pushes firing alerts to Alertmanager. Alertmanager then deduplicates, groups related alerts, applies silences and inhibition, and routes notifications to receivers such as Slack, PagerDuty, or email per its routing tree.

COMMON WRONG ANSWERS Omitting for: 5m causes pages on every brief rollout dip. Thinking Prometheus itself sends Slack messages; it only evaluates and forwards to Alertmanager, which notifies. Comparing the wrong metrics, like using replicas desired against itself.

LIKELY FOLLOW-UPS Difference between pending and firing states. How Alertmanager grouping and inhibition work. Why for protects against flapping.

ONE CONCRETE EXAMPLE An alert KubeDeploymentReplicasMismatch with expr available < desired and for 5m stays pending during a 90-second rollout, never paging, but fires if a deployment is stuck under-replicated for six minutes, routing a PagerDuty page via Alertmanager.