Executable Runbooks: Code, Not Just Checklists
An executable runbook turns a procedural document into an automated script. Instead of reading steps, you run them. It's used for incident response or maintenance, ensuring consistency. The footgun is not making them idempotent, which can worsen an outage.
WHY IT EXISTS: Traditional runbooks are static documents like wiki pages. During a high-stress incident, humans make mistakes: they misread steps, mistype commands, or skip a crucial check. This leads to longer outages and more errors. Executable runbooks were created to remove this human error factor for routine procedures.
THE MENTAL MODEL: Think of it as the difference between a recipe on a napkin and a pre-programmed bread machine. The recipe requires you to read, measure, and execute each step manually, with a high chance of error. The bread machine just needs you to press 'start' to get a consistent result. An executable runbook is the bread machine for system operations.
HOW IT WORKS: An executable runbook is a script (e.g., Python, Bash) or a workflow in a tool like Rundeck or a Jupyter Notebook. It encodes the logic from a traditional runbook. Instead of a document saying 'Step 1: SSH into server X,' the script contains the actual command to do so. It includes checks and balances, like verifying a service is down before trying to restart it, and provides clear output on its progress.
WHEN TO USE IT: Use them for frequent, well-understood, and high-stakes procedures. Three key areas are: first, incident response for common alerts (e.g., 'disk full' or 'service unresponsive'); second, routine maintenance (e.g., certificate rotation, database vacuuming); third, safe and repeatable deployments or rollbacks.
WHEN NOT TO USE IT: Avoid automating procedures that require nuanced human judgment or are not fully understood. If the diagnostic path for an issue is 'it depends,' writing a script is dangerous as it could take a destructive action based on a false premise. Always start with a manual runbook and only automate it once the procedure is stable.
ONE CANONICAL EXAMPLE: An alert for an unresponsive web server. A traditional runbook has manual steps to check and restart the process. An executable runbook is a script that automates this: it pings the health check, and if it fails, it SSHs to the machine, checks the process, attempts a graceful restart, and pages a human only if it cannot recover the service, logging every action along the way.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.