Eliminating toil

What toil is, and what it is not

In everyday speech, toil means any work you dislike. In SRE it has a specific meaning. The Google SRE book defines toil as work tied to running a production service that tends to have the following characteristics. The more of them a task has, the more clearly it is toil.

The last characteristic is the dangerous one. Work that grows in step with the service will eventually consume any team, however large, and hiring only buys time.

Two things are often confused with toil. Overhead is administrative work not tied to running a service: meetings, planning, reviews, training, expenses. It may be tedious, but it is not toil. Engineering work is the opposite of toil: it requires human judgement, produces a permanent improvement, and lets the team handle a larger service with the same effort. Writing the automation is engineering. Running the automation by hand every Tuesday is toil.

Why it matters

Some toil is unavoidable, and a little can even be pleasant: quick, predictable tasks give a sense of progress. The problem is quantity. The SRE guideline is that toil, together with other operational work such as on-call, should stay below 50% of each engineer's time, leaving at least half for engineering projects. That number is a ceiling, not a target.

• It crowds out the fix. Every hour spent restarting the worker is an hour not spent finding out why it hangs. Teams above the cap get slower at everything, permanently.
• It grows. Because toil scales with the service, a team that is comfortable today is underwater after a year of growth.
• It causes errors. Humans performing repetitive manual procedures make mistakes. A large share of outages trace back to a manual step done slightly wrong.
• It damages careers and morale. Engineers who spend their time on chores do not develop, become frustrated, and leave. The people who remain inherit more toil.
• It sets a precedent. A team that reliably absorbs manual work gets handed more of it, and developers lose the incentive to build systems that do not need it.

The cap is what makes the SRE model sustainable. When a team exceeds it, the response is structural: redirect some operational load back to the development team, pause new service onboarding, or dedicate a sprint to the largest sources. Simply working harder is not on the list.

Measuring toil

You cannot manage what you have not measured, and intuition about toil is poor: people underestimate frequent small interruptions and overestimate the rare dramatic ones. Measurement does not need to be elaborate.

1. Tag the work. Add a toil label in your ticket system, and a category such as cert-renewal, access-request, manual-deploy, disk-cleanup. Include on-call interrupts.
2. Record time roughly. Fifteen-minute granularity is plenty. Include the cost of the interruption itself: a five-minute task that breaks your concentration costs far more than five minutes.
3. Survey the team. Once a quarter, ask everyone to estimate their split between toil, on-call, overhead and engineering. It takes two minutes and the trend is what matters.
4. Aggregate by category and rank by total hours per month, not by how annoying each task feels.
5. Review monthly against the cap, and pick the top one or two categories to attack.

import csv
from collections import defaultdict

# toil_log.csv columns: date,engineer,category,minutes
TEAM_SIZE = 5
WORK_MINUTES_PER_MONTH = 160 * 60          # one engineer

minutes_by_category: dict[str, int] = defaultdict(int)
with open("toil_log.csv", newline="") as f:
    for row in csv.DictReader(f):
        minutes_by_category[row["category"]] += int(row["minutes"])

total = sum(minutes_by_category.values())
share = total / (TEAM_SIZE * WORK_MINUTES_PER_MONTH)
print(f"Toil: {total / 60:.0f} hours this month = {share:.0%} of team capacity\n")

for category, minutes in sorted(minutes_by_category.items(), key=lambda kv: -kv[1]):
    print(f"{category:<20} {minutes / 60:6.1f} h   {minutes / total:5.0%} of toil")

The output almost always follows a Pareto pattern: two or three categories account for most of the hours. That is good news, because it tells you exactly where a small amount of engineering will return the most time.

Is it worth automating?

Not everything should be automated. Automation is software: it has to be written, tested, documented, secured, monitored and maintained, and it can fail in ways a human would not. A simple comparison keeps you honest.

time saved per year = occurrences per year x minutes per occurrence

payback period      = build cost / time saved per year

Example: certificate renewal
  occurrences        40 per year
  time each          45 minutes (including the context switch)
  time saved         40 x 45 = 1,800 minutes = 30 hours per year
  build cost         16 hours to set up automated renewal
  maintenance        about 2 hours per year
  payback            16 / (30 - 2) = 0.57 years, about 7 months

Time saved is only part of the value, and often the smaller part. Weigh these as well.

• Consistency. Automation performs the step identically every time. If a mistake in the manual procedure could cause an outage, as an expired certificate does, that alone can justify the work.
• Speed. A machine reacts in seconds. For anything on an incident's critical path, such as failover, that matters more than the hours saved.
• Growth. A task that happens 40 times this year may happen 400 times next year. Decide using where the service is going.
• Interruption cost. Removing a task that arrives at random times returns more focus than its minutes suggest.
• Frequency of change. If the procedure changes every month, the automation will be permanently out of date. Stabilise the process first.

Before automating anything, ask a better question: can the task be eliminated? Disk cleanup disappears if logs are shipped off the machine with a retention policy. Access requests disappear if access derives from group membership in the identity provider. Manual deploys disappear with the pipelines from the DevOps track. Removing the need is cheaper and more reliable than automating the response, and it never needs maintenance.

From runbook to autonomous system

Automation is not all or nothing. The SRE book describes a progression, and moving up even one level is a win. The example is a database failover.

Level 1 is a real and underrated step. Writing the procedure down makes it repeatable, lets anyone perform it, and is the specification for the script that follows. Do not skip it.

Automation acts faster and at a greater scale than any human, which includes making mistakes faster and at a greater scale. Build the safety in from the start.

#!/usr/bin/env bash
# Remove build caches older than N days. Safe to run repeatedly.
set -euo pipefail

days="${1:-14}"
target="/var/cache/builds"
dry_run="${DRY_RUN:-true}"                     # safe by default

[[ -d "$target" ]] || { echo "no such directory: $target" >&2; exit 1; }

count="$(find "$target" -mindepth 1 -maxdepth 1 -mtime +"$days" | wc -l)"
echo "$(date -u +%FT%TZ) candidates=$count older_than_days=$days dry_run=$dry_run"

if (( count > 500 )); then                     # sanity limit: refuse surprising work
  echo "refusing: $count entries exceeds the safety limit" >&2
  exit 2
fi

if [[ "$dry_run" == "false" ]]; then
  find "$target" -mindepth 1 -maxdepth 1 -mtime +"$days" -exec rm -rf {} +
fi

• Idempotent, so that running it twice, or after a partial failure, does no harm.
• Dry run by default for anything destructive, with the real action requiring an explicit flag.
• Sanity limits. Automation that is about to touch far more than expected should stop and ask. Many famous outages were scripts faithfully applying a bad input to the whole fleet.
• Logged and observable. Record what it did and why, and alert when it fails or has not run.
• Gradual. Roll automated changes across a fleet in stages, as you would a release.
• Keep the manual path alive. People must still be able to do the task when the automation is broken, so keep the runbook current and practise it occasionally.