Auto-Remediation: Automated Fixes for Common Failures
Auto-remediation is a system's immune response, automatically detecting and fixing known problems like a crashed service. It's a core SRE practice for improving availability, but a bad script can create a 'remediation storm' that worsens an outage.
Why it exists
Human intervention is a bottleneck. For recurring, well-understood system failures, waiting for an on-call engineer to wake up, diagnose, and act increases downtime. To meet strict reliability targets, SRE principles favor automating these responses.
The mental model
Think of it as a runbook that a machine executes. Instead of a person following a checklist to restart a service, a script does it instantly upon detecting a specific alert. It's the programmatic evolution of "have you tried turning it off and on again?", applied at scale.
How it works
Auto-remediation is a closed loop. First, a monitoring system detects an alert-worthy condition, like a service health check failing. Second, an alert fires to an automation platform. Third, that platform executes a pre-defined script or action—the remediation—such as restarting a process or clearing a cache. Finally, the system verifies the fix and resolves the alert, often without a human ever being paged.
When to use it
Use it for frequent, predictable, and low-risk failures. Good candidates are restarting a stateless service that has crashed, clearing temporary files when a disk is nearing capacity, or scaling up a server fleet in response to a sustained, non-malicious traffic increase. The problem and its solution must be well-understood.
When not to use it
Avoid auto-remediation for complex, novel, or high-risk incidents. If the fix could cause data loss or has unpredictable cascading effects (like restarting a core stateful database), human judgment is required. A poorly designed automated fix can turn a small problem into a major outage.
One canonical example
A web service running in containers has a known memory leak and crashes periodically. The monitoring system detects the container is unhealthy. Instead of paging an engineer, an alert triggers a script that automatically restarts the container. This brings a fresh instance online within seconds, resolving the immediate user-facing impact while engineers work on a permanent fix for the leak.
Interview question
Which scenario best exemplifies a suitable candidate for implementing auto-remediation?
- a.A sudden, unexplained surge in traffic that could indicate a security breach.
- b.A stateless microservice occasionally crashing due to a known memory leak.Correct
- c.A newly discovered, intermittent bug causing a critical database to corrupt data.
- d.A complex network routing issue requiring expert analysis to diagnose.
Why? this is the answer
Auto-remediation is best suited for frequent, predictable, and low-risk failures with well-understood solutions, such as restarting a stateless service with a known issue. Complex, novel, or high-risk incidents like data corruption or potential security breaches require human judgment.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on sre — each one lists the topics its interview covers.
See open roles