Runbooks & operational readiness

What a runbook is

A runbook, sometimes called a playbook, is a set of instructions for handling one specific operational situation: responding to a particular alert, performing a failover, restoring from backup, rotating a credential. It is written for the person who has to act, in the order they need to act.

It is not architecture documentation, which explains how the system works and is read at leisure. A runbook may link to that, but it must not make the reader wade through it. And it is not automation, although it is often the step just before it: the toil module described the ladder from "no procedure" to "documented procedure" to "script" to "automatic". A good runbook is the specification for the script that will eventually replace it.

The Google SRE book reports that thinking through and recording best practice ahead of time in a playbook gives roughly a threefold improvement in time to repair, compared with improvising. The reason is simple. Under stress people lose working memory and fall back on what is in front of them, so make sure that something good is in front of them.

Anatomy of an alert runbook

# CheckoutErrorBudgetFastBurn
Owner: checkout team (#team-checkout)     Last tested: 2026-08-12 (game day)
Severity: page     Dashboard: https://grafana.example.com/d/checkout

## What this means
Checkout is failing requests fast enough to spend 2% of its monthly error
budget in an hour (error ratio above 1.44%). Customers cannot complete
purchases. Treat as SEV2 unless the error ratio is above 25%, then SEV1.

## First: check for a recent change (2 minutes)
1. Open the dashboard. Is there a deploy annotation within 15 minutes of the
   start of the errors?
2. If yes, ROLL BACK NOW. Do not investigate first.

       kubectl rollout undo deployment/checkout -n shop
       kubectl rollout status deployment/checkout -n shop

   Expected: error ratio starts falling within 3 minutes.
   If it does not fall, continue below.

## Diagnose
3. Which routes are failing? Dashboard row 'Breakdown', panel 'Errors by route'.
4. Is one dependency responsible? Row 'Dependencies'.
   - payment-provider errors high -> see runbook payment-provider-down.md
   - database latency high        -> see runbook checkout-db-saturated.md
5. Are the pods healthy?

       kubectl get pods -n shop -l app=checkout
       kubectl logs deploy/checkout -n shop --since=10m | grep -i error | tail -30

## Mitigations
- Dependency down: enable degraded mode (orders are queued, confirmed later):

       kubectl set env deployment/checkout -n shop CHECKOUT_DEGRADED_MODE=true

  REMEMBER to switch it off afterwards. Record it in the incident channel.
- Saturated: scale out.  kubectl scale deployment/checkout -n shop --replicas=12

## Escalate
No progress after 20 minutes, or error ratio above 25%: page the checkout
secondary, then the platform on-call (database, networking).

## Afterwards
Undo temporary changes. Open a postmortem if any budget threshold was crossed.

• Meaning and impact first. The responder needs to know at once how bad it is, in users' terms, and what severity to declare.
• The most likely fix comes first, before the diagnosis. Most incidents follow a change, so the rollback is step one and not the conclusion of an investigation.
• Commands are exact and can be pasted, with real namespaces and names. Never "restart the service"; always the command that restarts it.
• Say what to expect after each action, and what to do if that does not happen. Without it, the reader cannot tell whether the step worked.
• Branches point to other runbooks instead of growing into one enormous document.
• Escalation is explicit: when, and to whom.
• Temporary changes are flagged for reversal at the point where they are made.

Writing for a tired stranger

Write for someone who is competent in general but does not know this service, is reading on a laptop at night, and is under pressure. That reader is more common than you think: the newest member of the rotation, an engineer covering for a colleague, or you in a year's time.

• Numbered steps, one action each, in the imperative. "Check the queue depth", not a paragraph describing the queueing subsystem.
• Put decision points in plain sight: "If X, go to step 6. Otherwise continue."
• Mark danger clearly. Anything destructive or irreversible gets a warning immediately before the command, stating what it destroys and how to confirm the target first.
• Link directly to the exact dashboard, log query or console page, not to a home page from which the reader must navigate.
• No secrets. State where a credential is stored and how to obtain access, never the credential.
• Explain why, briefly, where it helps judgement. A reader who understands that degraded mode queues orders can decide whether it fits a situation the author did not foresee.
• Keep it short. A runbook that takes ten minutes to read will not be read. Move background into linked documents.

A runbook is a guide for a thinking person and not a script for a machine. Real incidents seldom match the expected pattern exactly, so the best runbooks combine concrete steps for the common cases with enough explanation that the reader can adapt. If a procedure truly needs no judgement at all, it should not be a runbook for long. It should be automated.

Store runbooks where responders can find them in seconds: in a repository alongside the code, rendered somewhere searchable, with predictable names that match the alert names. Make sure they are reachable when the thing they describe is broken. A runbook for a wiki outage that lives only on the wiki is a classic trap.

Keeping runbooks alive

A wrong runbook is worse than none, because the reader trusts it. Commands refer to a cluster that was renamed, a dashboard that was deleted, a flag that no longer exists. Runbooks decay at the same speed as the system changes, so they need the same care as code.

• Own them. Each runbook names an owning team. When the team changes a service, updating its runbooks is part of the change, reviewed in the same pull request.
• Date them. Record when the runbook was last tested, as distinct from last edited. An old date is a warning to the reader and a prompt to the owner.
• Test them. Follow the runbook step by step during game days and drills, ideally with someone who did not write it. Every place where they hesitate or have to ask is a defect.
• Fix them at the moment of use. The on-call engineer who finds an error corrects it that day or files a ticket. Handoff notes and postmortems both have a standing section for runbook problems.
• Link from the alert, through the runbook_url annotation. A CI check can fail the build when an alert has no runbook link, or when the link is dead.
• Retire them. When a runbook has been fully automated, or its alert deleted, delete the runbook as well.

#!/usr/bin/env bash
set -euo pipefail

status=0
while IFS=$'\t' read -r alert url; do
  if [[ -z "$url" || "$url" == "null" ]]; then
    echo "MISSING runbook_url: $alert" >&2; status=1; continue
  fi
  file="runbooks/${url##*/runbooks/}.md"
  [[ -f "$file" ]] || { echo "DEAD LINK: $alert -> $file" >&2; status=1; }
done < <(yq -r '.groups[].rules[]
               | select(.alert != null and .labels.severity == "page")
               | [.alert, .annotations.runbook_url] | @tsv' rules/*.yml)
exit "$status"

The production readiness review

Runbooks cover what to do when things go wrong. A production readiness review (PRR) asks, before launch, whether the service can be operated at all. It began at Google as the gate a service had to pass before an SRE team would take on its pager, and it is just as useful for a team that runs its own services: a structured conversation, guided by a checklist, between the people who built the service and the people who will be woken by it.

Treat the outcome as a list of risks, not a pass or fail. Each gap is either fixed before launch, or accepted explicitly by a named person, with a date by which it will be fixed. That is risk management as the embracing-risk module described it: a deliberate decision, made in daylight, and written down.

Scale the process to the service. A new tier-one payment system deserves a formal review with several teams. An internal tool needs fifteen minutes with the checklist. The best long-term investment is to make the checklist unnecessary by building the answers into the platform: a service template that arrives with dashboards, SLO alerts, probes, a pipeline with canary rollout, and a runbook skeleton already in place, so that the ready path is also the easy one.

The minimum before real traffic

If time is short, these are the items that separate a survivable launch from a miserable one. Each takes hours, not weeks.

1. An owner and a pager. Someone knows they are responsible, and alerts reach them.
2. One SLO with a burn-rate alert on the main user journey, and a dashboard showing rate, errors and latency.
3. A rollback that has been performed for real, and timed, by someone other than its author.
4. Timeouts on every outgoing call, so that a slow dependency cannot hang the whole service.
5. Readiness and liveness probes, and graceful shutdown, so that deploys and node changes do not drop requests.
6. A restore from backup that has been tested, if the service holds data.
7. A runbook for each paging alert, however short, and an entry for the service in the on-call handbook.

kubectl get deploy checkout -n shop \
  -o jsonpath='{.spec.template.spec.containers[0].readinessProbe}{"\n"}'
kubectl get deploy checkout -n shop \
  -o jsonpath='{.spec.template.spec.containers[0].resources}{"\n"}'
kubectl get pdb,hpa -n shop                      # disruption budget and autoscaler present?
kubectl rollout history deployment/checkout -n shop   # is there a revision to roll back to?
kubectl auth can-i --list -n shop --as system:serviceaccount:shop:checkout | head

Readiness does not end at launch. Services change, traffic grows and teams turn over. Revisit the checklist yearly for important services, and whenever ownership moves to a new team. The next stage of this track turns to the engineering that makes the answers better: handling failure, planning capacity, releasing safely and testing resilience on purpose.