Auto-Remediation: Automated Fixes for Common Failures
Auto-remediation is a system's immune response, automatically detecting and fixing known problems like a crashed service. It's a core SRE practice for improving availability, but a bad script can create a 'remediation storm' that worsens an outage.
WHY IT EXISTS: Human intervention is a bottleneck. For recurring, well-understood system failures, waiting for an on-call engineer to wake up, diagnose, and act increases downtime. To meet strict reliability targets, SRE principles favor automating these responses.
THE MENTAL MODEL: Think of it as a runbook that a machine executes. Instead of a person following a checklist to restart a service, a script does it instantly upon detecting a specific alert. It's the programmatic evolution of "have you tried turning it off and on again?", applied at scale.
HOW IT WORKS: Auto-remediation is a closed loop. First, a monitoring system detects an alert-worthy condition, like a service health check failing. Second, an alert fires to an automation platform. Third, that platform executes a pre-defined script or action—the remediation—such as restarting a process or clearing a cache. Finally, the system verifies the fix and resolves the alert, often without a human ever being paged.
WHEN TO USE IT: Use it for frequent, predictable, and low-risk failures. Good candidates are restarting a stateless service that has crashed, clearing temporary files when a disk is nearing capacity, or scaling up a server fleet in response to a sustained, non-malicious traffic increase. The problem and its solution must be well-understood.
WHEN NOT TO USE IT: Avoid auto-remediation for complex, novel, or high-risk incidents. If the fix could cause data loss or has unpredictable cascading effects (like restarting a core stateful database), human judgment is required. A poorly designed automated fix can turn a small problem into a major outage.
ONE CANONICAL EXAMPLE: A web service running in containers has a known memory leak and crashes periodically. The monitoring system detects the container is unhealthy. Instead of paging an engineer, an alert triggers a script that automatically restarts the container. This brings a fresh instance online within seconds, resolving the immediate user-facing impact while engineers work on a permanent fix for the leak.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.