Distinguishing toil from necessary ops work
Defining toil precisely.
toil is manual, repetitive, automatable, tactical, and scales with growth; engineering and judgment-heavy work is not toil.
calling all ops work toil, or thinking toil means unimportant work.
WHAT THIS TESTS Whether you know the precise SRE definition of toil and can separate it from valuable operational engineering, so you automate the right things.
A GOOD ANSWER COVERS Toil has specific properties: it is manual, repetitive, automatable, tactical and interrupt-driven, devoid of lasting value, and it scales linearly with the size or traffic of the service. Work that is genuinely creative, requires human judgment, is novel, or produces a permanent improvement is not toil even if it is operational. The danger of toil is that it grows with the system and crowds out engineering time, so SRE caps it, often around half of time, and invests in automating it away. Not everything manual is toil: a one-time migration design, a careful capacity plan, or a never-seen-before incident investigation are valuable operational work that should not be blindly automated.
COMMON WRONG ANSWERS Calling all operations toil. Thinking toil means unimportant. Assuming any manual task must be automated regardless of frequency or value.
LIKELY FOLLOW-UPS How much toil is acceptable? How do you decide if automating is worth the cost? What if automating is harder than the toil itself?
ONE CONCRETE EXAMPLE Toil: every morning an engineer manually runs the same five commands to rotate logs and restart a stuck worker, a repetitive, automatable, value-less task that grows as you add servers, so you script and schedule it. Non-toil: planning next quarter's capacity, which requires forecasting, judgment, and tradeoff analysis, or investigating a brand-new failure mode, which is novel engineering. Both are operational, but only the first fits every property of toil and is worth automating; the second creates lasting value and needs a human.
Read the original → sre.google
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.