Toil: The Repetitive Work That Kills Engineering Velocity
Toil is manual, repetitive work that scales with your service, stealing time from real engineering. It's the operational treadmill of handling the same alerts or manually running scripts.
WHY IT EXISTS Engineers need a precise way to distinguish between valuable, long-term project work and the operational load that prevents it. The concept of "toil" was created to define, measure, and ultimately eliminate the kind of work that keeps teams reacting instead of building.
THE MENTAL MODEL If a human operator needs to touch your system during normal operations, you have a bug. Toil is any operational task that is manual, repetitive, automatable, tactical, has no lasting value, and scales linearly with the service. The goal is to cap this work at less than 50% of an engineer's time, freeing them for permanent improvements.
HOW IT WORKS Toil is identified by a set of specific attributes. A task is likely toil if it is: Manual (even running a script by hand counts), Repetitive (you do it over and over), Automatable (a machine could do it or the need could be designed away), and Tactical (interrupt-driven and reactive, like a pager alert). Critically, after the work is done, the service is in the exact same state as before; there is no enduring value created.
WHEN TO USE IT Use the concept of toil to audit your team's workload. Categorize tasks to see if they fit the profile: manually restarting a flaky service, hand-editing a config to add capacity, or running a report by hand every week. If a task's workload grows proportionally as your user count or traffic grows, it is a primary candidate for being labeled toil and targeted for automation.
WHEN NOT TO USE IT Do not label all unpleasant or administrative work as toil. Team meetings, HR paperwork, and writing quarterly goals are overhead, not toil. Similarly, one-off "grungy" work that provides a permanent improvement, like a major cleanup of alerting configurations, is valuable engineering, not toil. If a task requires human judgment and is not repetitive, it is not toil.
ONE CANONICAL EXAMPLE Manually running a script to provision resources for a new customer is pure toil. The task is manual, repetitive (you do it for every new customer), and completely automatable. It is tactical, reacting to a new signup, and offers no enduring value to the system itself. The work scales linearly: 100 customers requires running the script 100 times. This work should be replaced by an automated provisioning system, which is a one-time engineering project with lasting value.
Read the original → sre.google
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.