More in DevOps & Cloud — page 8
Postmortem Report Template
A postmortem template is a standard structure for documenting an incident: summary, impact, timeline, root cause, what went well and poorly, and action items.
Incident Timeline Reconstruction
Timeline reconstruction is the after-the-fact process of merging evidence from logs, metrics, deploys, and chat into one accurate sequence of what happened.
War Room in Incident Response
A war room is a dedicated space, physical or virtual, where responders coordinate during a major incident. It centralizes communication and decision-making under a defined incident commander, cutting confusion and duplicated effort, but should be reserved for…
Runbooks
A runbook is a documented, step-by-step procedure for handling a specific operational task or known failure, such as responding to an alert. It captures expert knowledge so any on-call engineer can act quickly and consistently, reducing reliance on tribal…
Incident Timeline
An incident timeline is a chronological, timestamped record of what happened during an incident: detection, key events, actions taken, and resolution. It anchors the postmortem in facts, separates symptoms from causes, and reveals detection and response…
Baggage in Distributed Tracing
Baggage is key-value context propagated alongside a trace across service boundaries, so downstream services can read values set upstream. It enables cross-cutting context like tenant id, but it travels in headers on every hop, so overuse adds latency and leak…
Auto-Instrumentation
Auto-instrumentation automatically adds telemetry to an application without manual code changes, by hooking into libraries, frameworks, or the runtime. It gives broad baseline observability fast, but produces generic spans that often need manual…
Metrics in Observability
Metrics are numeric measurements aggregated over time, like counters, gauges, and histograms. They are cheap to store and fast to query, making them ideal for dashboards and alerting, but their pre-aggregation discards per-event detail needed for deep…
Observability vs Monitoring
Monitoring watches predefined metrics and alerts on known failure modes you anticipated. Observability is the property of being able to ask new questions about a system's internal state from its outputs, letting you debug unknown failures you never predicted…
Alerting on under-replicated Deployments
WHAT IT TESTS: Prometheus alerting pipeline. OUTLINE: write an alerting rule comparing kube_state_metrics available vs desired replicas with for: 5m, Prometheus evaluates and fires to Alertmanager, which dedupes/routes/notifies.
PromQL for top 5 CPU-consuming pods
WHAT IT TESTS: PromQL on counters. OUTLINE: apply rate() to the counter over 15m, sum by pod to combine containers, then wrap in topk(5); rate handles counter resets.
EFK centralized logging architecture
WHAT IT TESTS: End-to-end log pipeline design. OUTLINE: Fluentd runs as a DaemonSet collecting node container logs, parses and forwards to Elasticsearch for indexed storage, and Kibana queries and visualizes them.
Auto-discovering app pods for Prometheus scraping
WHAT IT TESTS: Kubernetes service discovery in Prometheus. OUTLINE: use kubernetes_sd_configs with role pod, relabel on pod annotations like prometheus.io/scrape to filter, and set path and port; with the Operator use a PodMonitor or ServiceMonitor.
kube-state-metrics versus node-exporter
WHAT IT TESTS: Distinguishing object-state from host metrics. OUTLINE: kube-state-metrics exposes API object state (deployment replicas, pod phase, restarts) from the control plane, while node-exporter exposes OS-level hardware metrics (CPU, memory, disk) per…
Viewing pod logs and durable log collection
WHAT IT TESTS: Logging basics and durability. OUTLINE: kubectl logs (with -c, --previous, -f) reads container stdout/stderr; because that storage is ephemeral, run a node-level logging agent as a DaemonSet shipping logs to a central store.
Multi-tenant isolation with a monitoring exception
WHAT IT TESTS: Layered NetworkPolicy design. OUTLINE: apply default-deny ingress per tenant namespace, allow same-namespace traffic, then add an ingress rule permitting the monitoring namespace via namespaceSelector on the metrics port.
Binding a ClusterRole with a RoleBinding
WHAT IT TESTS: Reusing a ClusterRole at namespace scope. OUTLINE: a RoleBinding referencing a ClusterRole grants those rules only within the binding's namespace; reuse built-in roles like view per-team without duplicating definitions.
Root and NET_ADMIN under Pod Security Standards
WHAT IT TESTS: securityContext plus admission policy. OUTLINE: set runAsUser 0 and capabilities add NET_ADMIN in the container securityContext; this is rejected by Restricted and Baseline, so the namespace must use the Privileged profile.
Write a frontend-to-backend NetworkPolicy
WHAT IT TESTS: Authoring a correct ingress NetworkPolicy. OUTLINE: set podSelector to app=backend, policyTypes Ingress, one ingress rule with from podSelector app=frontend and ports TCP 8080; the implicit deny handles the rest.
Namespace-scoped RBAC for a ServiceAccount
WHAT IT TESTS: Designing least-privilege namespaced access. OUTLINE: create a Role in production granting create on deployments (apps group) and services (core group), then a RoleBinding tying that Role to the ServiceAccount.