Run a full incident drill

A page from Alertmanager arrives on your phone within a minute, with the runbook link, and escalates to a second route if nobody acknowledges.

1. 1.1
   Create the branch and the directory layout, and copy the SLO project's runbook into incidents/runbooks/ so everything the responder needs lives in one place.

   bashCopy

   cd zero-to-prod && git switch -c feat/incidents
   mkdir -p incidents/roles incidents/templates incidents/runbooks chaos/scenarios deploy/relay
   cp slo/RUNBOOK-orders-api.md incidents/runbooks/orders-api.md

2. 1.2
   Pick a channel that reaches your phone. For a free, no-account option use ntfy: install the app, subscribe to a topic with an unguessable name, and confirm a test push arrives.

   bashCopy

   TOPIC="oncall-$(openssl rand -hex 6)"; echo "$TOPIC"
   curl -s -d "test page from the drill setup" "https://ntfy.sh/$TOPIC"
   # subscribe to $TOPIC in the ntfy app; the message should appear within seconds

   Slack or Discord incoming webhooks work the same way with a different URL; PagerDuty or Opsgenie free tiers add proper acknowledgement and escalation if you want the real thing.
3. 1.3
   Alertmanager cannot post to ntfy's simple API directly with the right headers, so add a tiny relay next to the alert sink that turns an Alertmanager webhook into a push with title, priority and the runbook link.

   pythonCopy

   # deploy/relay/relay.py  (run in a small python:3.12-slim container)
   import json
   import os
   import urllib.request
   from http.server import BaseHTTPRequestHandler, HTTPServer
   
   TOPIC = os.environ["NTFY_TOPIC"]
   
   
   class H(BaseHTTPRequestHandler):
       def do_POST(self):
           body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
           for a in body.get("alerts", []):
               firing = a["status"] == "firing"
               title = ("FIRING: " if firing else "RESOLVED: ") + a["labels"].get("alertname", "?")
               msg = a["annotations"].get("summary", "") + "\n" + a["annotations"].get("runbook", "")
               req = urllib.request.Request(f"https://ntfy.sh/{TOPIC}", data=msg.encode(), method="POST")
               req.add_header("Title", title)
               req.add_header("Priority", "urgent" if firing and a["labels"].get("severity") == "page" else "default")
               req.add_header("Tags", "rotating_light" if firing else "white_check_mark")
               urllib.request.urlopen(req, timeout=10)
           self.send_response(200); self.end_headers()
   
   
   HTTPServer(("0.0.0.0", 8090), H).serve_forever()

   Compose: a service relay built from a two-line Dockerfile (FROM python:3.12-slim, COPY relay.py /relay.py, CMD ["python", "/relay.py"]) with NTFY_TOPIC from .env. Never commit the topic name; it is effectively a secret.
4. 1.4
   Add the relay to the Compose stack and validate the Alertmanager configuration before restarting it.

   yamlCopy

   # deploy/compose.yml (addition)
   services:
     relay:
       build: ./relay
       environment:
         NTFY_TOPIC: ${NTFY_TOPIC}
       restart: unless-stopped

   Put NTFY_TOPIC=... in deploy/.env (git-ignored). Validate with docker run --rm -v "$PWD/deploy/alertmanager.yml:/a.yml:ro" --entrypoint amtool prom/alertmanager:v0.28.0 check-config /a.yml, then docker compose -f deploy/compose.yml up -d --build relay alertmanager.
5. 1.5
   Route pages to the relay with an escalation: if the same alert is still firing after 15 minutes, a second route with a louder priority fires. Alertmanager has no acknowledgement concept, so escalation is modelled by repeat_interval on a second receiver plus a time-based route; for real acknowledgement, a pager product is the answer, and the drill notes that gap.

   yamlCopy

   # deploy/alertmanager.yml (routes section)
   route:
     receiver: tickets
     group_by: [alertname, service]
     group_wait: 10s
     group_interval: 2m
     repeat_interval: 4h
     routes:
       - matchers: [severity="page"]
         receiver: oncall-primary
         continue: true
         repeat_interval: 15m            # re-pages every 15 min while firing
       - matchers: [severity="page"]
         receiver: oncall-escalation
         group_wait: 15m                 # first notification only after 15 min of firing
         repeat_interval: 30m
   
   receivers:
     - name: oncall-primary
       webhook_configs: [{url: http://relay:8090/, send_resolved: true}]
     - name: oncall-escalation
       webhook_configs: [{url: http://relay:8090/escalation, send_resolved: true}]
     - name: tickets
       webhook_configs: [{url: http://alert-sink:8080/tickets, send_resolved: true}]

   Extend the relay to treat the /escalation path as priority max with a different tag, and to include ESCALATED in the title.
6. 1.6
   Prove the page path with the SLO project's chaos variable, then reset. Note the time from injection to the push arriving; that is your alerting latency and it belongs in the process doc.

   bashCopy

   date -u && CHAOS_ERROR_RATE=0.05 docker compose -f deploy/compose.yml up -d app
   # wait for the push; then
   CHAOS_ERROR_RATE=0 docker compose -f deploy/compose.yml up -d app

   Check: A push titled FIRING: OrdersApiAvailabilityFastBurn arrives with the runbook URL; a RESOLVED push follows after the fix.
7. 1.7
   Test the escalation path deliberately: leave the injected fault in place for twenty minutes and confirm the second, louder push arrives at the fifteen-minute mark. Then reset. Record both latencies (first page, escalation) in PROCESS.md under "alerting latency".

   bashCopy

   date -u && CHAOS_ERROR_RATE=0.05 docker compose -f deploy/compose.yml up -d app
   # wait ~20 minutes; expect the ESCALATED push at about +15 min
   CHAOS_ERROR_RATE=0 docker compose -f deploy/compose.yml up -d app

   Check: Two pushes with different titles and priorities, roughly fifteen minutes apart.

Severity levels, roles, declare and resolve criteria, a comms cadence and the templates — short enough that people actually use them at 3 a.m.

1. 2.1
   Write PROCESS.md. Keep it to one screen per section. Severities decide who is woken and how often updates go out; roles decide who talks and who types.

   textCopy

   # incidents/PROCESS.md
   
   ## Severities
   SEV1  customer-facing outage or data loss; error budget burning >14x     page now; updates every 30 min; postmortem required
   SEV2  degraded (latency SLO burning, partial feature outage)             page in hours; updates hourly; postmortem required
   SEV3  internal-only or single-customer impact; workaround exists          ticket; next business day; postmortem optional
   
   ## Declare when
   - a page fires and is confirmed real, or
   - anyone believes customers are affected. Declaring is cheap; do it early. Downgrade later if wrong.
   
   ## Roles (one person can hold several; every role is named in the channel)
   Incident Commander (IC)  owns decisions, keeps the timeline moving, decides mitigation vs investigation, calls resolve
   Investigator             hands on the system; reports findings to the IC; does not communicate outward
   Communications           writes status updates on the cadence; answers stakeholders so nobody else has to
   Scribe                   records every observation, decision and action with a UTC timestamp
   
   ## During
   - mitigate before you diagnose: rollback, feature flag, scale, failover; investigation continues after
   - one channel, one timeline, one IC; hand over roles explicitly ("you have IC") and log it
   - no speculation in status updates: what we know, what we are doing, when the next update is
   
   ## Resolve when
   - the SLI is back within target for 15 minutes and the mitigation is stable, then
   - IC posts the resolution and the postmortem owner
   
   ## After
   - postmortem draft within 2 business days, review within 5, action items with owners and dates
   - blameless: we fix the system that allowed the failure, not the person who touched it

2. 2.2
   Decide and write down the on-call expectations that the process assumes: response time to a page (for example 10 minutes), how handovers happen, and what compensation or time off applies. For a solo project this is one paragraph, but it belongs in the file because the numbers in the drill are measured against it.

   textCopy

   # incidents/PROCESS.md (on-call section)
   ## On-call
   - one primary, one escalation contact; rotation weekly; handover Monday 10:00 with a written summary of open issues
   - expected: acknowledge a page within 10 minutes, be at a keyboard within 20
   - a paged night earns time off the next morning; more than two pages a week triggers a review of the alert, not the person

3. 2.3
   Write the role cards: one file each with the three things that role does first, what it must not do, and the handover phrase. The scribe's card includes the timeline format.

   textCopy

   # incidents/roles/scribe.md
   First three things:
     1. Open incidents/templates/timeline.md as incidents/<date>-<name>/timeline.md; record the page time and who acknowledged.
     2. Pin the channel message "Timeline: <link>".
     3. Log every observation, decision, action and handover as it happens, with `date -u` timestamps. Ask "can you say that in one line for the timeline?"
   
   Do not: investigate, fix, or communicate outward. If you are alone, you are IC and scribe; the timeline still gets written.
   
   Format:
     HH:MM:SSZ  [who]  observation | decision | action | handover  - text
     10:02:14Z  [alert]  action  - FIRING OrdersApiAvailabilityFastBurn, page delivered
     10:03:40Z  [alice]  observation - dashboard: 503s from all pods since 09:58; no deploy since yesterday
     10:05:02Z  [alice]  decision - IC: mitigate first; rolling back to previous digest
   
   Handover: "You have scribe." / "I have scribe."

   Write commander.md, investigator.md and communications.md the same way. The comms card holds the status-update template and the rule that the first update goes out within 15 minutes of declaring, even if it only says "we are investigating".
4. 2.4
   Write the two templates that get used under pressure: the status update and the postmortem skeleton.

   textCopy

   # incidents/templates/status-update.md
   [SEVn] <service> - <one-line impact> - update #<n> at <HH:MM UTC>
   What we know:   <facts only>
   What we are doing: <current mitigation / investigation>
   Customer impact: <who, what, since when>
   Next update:    <time, at most the cadence for this severity>
   IC: <name>   Channel: <link>
   
   # incidents/templates/postmortem.md
   # <date> <service>: <title>
   Owner:            Reviewed:            Severity:            Duration (impact):
   
   ## Summary (3 sentences a director can read)
   ## Impact (numbers: error budget consumed, requests failed, customers affected)
   ## Timeline (from the scribe's file; keep it factual)
   ## Root cause and contributing factors (the system, not the person)
   ## What went well / What went poorly / Where we got lucky
   ## Detection: time from start of impact to page; what would have caught it sooner
   ## Action items
   | # | action | owner | due | type (prevent / detect / mitigate / process) | status |
   ## Lessons for other teams

5. 2.5
   Extend the runbooks from the SLO project with the three mitigations the drill may need, each as an exact command: roll back to the previous image, scale out, and disable the chaos variables. A mitigation you have to look up during an incident is not a mitigation.

   bashCopy

   # incidents/runbooks/orders-api.md (mitigations section)
   # 1. roll back to the previous known-good image (digest recorded at every deploy in deploy/RELEASES.md)
   sed -i 's#^APP_IMAGE=.*#APP_IMAGE=ghcr.io/YOUR_GITHUB_USER/zero-to-prod@sha256:PREVIOUS#' deploy/.env
   docker compose -f deploy/compose.yml up -d app
   
   # 2. scale out (local stack: more app replicas behind the compose network; production: the ASG / HPA)
   docker compose -f deploy/compose.yml up -d --scale app=3 app
   
   # 3. remove injected faults (the drill's favourite)
   sed -i 's/^CHAOS_.*//' deploy/.env && docker compose -f deploy/compose.yml up -d app

   Add deploy/RELEASES.md and have the release workflow append a line with the date and digest on every deploy; the rollback command needs it.
6. 2.6
   Commit the process, and brief your breaker: give them the chaos/README.md with the rules (what they may break, what they must not, how to restore if the drill goes wrong) and three or four scenario ideas — but let them choose and keep it secret.

   textCopy

   # chaos/README.md (for the breaker)
   You may: change env vars in deploy/.env, stop/restart containers, edit deploy/prometheus.yml or the rules,
           change the Grafana datasource, fill the disk with a large file, add latency with `tc` inside the app container.
   You must not: delete volumes, delete the repo, touch the laptop's network.
   Scenarios (pick one, do not tell the responder which):
     A. CHAOS_ERROR_RATE=0.08 (clear signal, tests the page path and rollback reflex - they will look for a deploy)
     B. CHAOS_SLOW_RATE=0.6 + stop prometheus for 10 min (latency burn AND blind dashboards: tests recovery of observability)
     C. edit deploy/rules/slo_alerts.yml to break the fast-burn expression, then A (tests noticing the missing page)
     D. `docker exec app tc qdisc add dev eth0 root netem delay 400ms` (needs NET_ADMIN; tests investigation without a config change)
   Restore: git checkout deploy/ && docker compose -f deploy/compose.yml up -d --force-recreate

   Check: The process files are merged; the breaker has the README and a time window; you have the ntfy app on and your role cards open.
7. 2.7
   Dry-run the process with a tabletop before the live drill: read a scenario aloud, and have each role say what they do first, using only the cards. Fix anything that needed explaining; the cards should be enough.

   Check: Each role card answers 'what do I do first' without discussion; the status-update template can be filled in under a minute.

Respond to an unknown fault end to end with the process, producing a complete timeline and hard numbers for detection, mitigation and resolution.

1. 3.1
   Start the load generator so the SLIs have traffic, then step away. The breaker acts sometime in the next hour without telling you.

   bashCopy

   hey -z 90m -q 20 -c 2 http://localhost/orders/42 > /dev/null 2>&1 &

2. 3.2
   When the page arrives: acknowledge in the channel ("ack, looking"), open the timeline from the template, record the page time, and declare. If you are alone, you are IC, investigator and scribe; say so in the timeline.

   textCopy

   10:02:14Z  [alert]   action     - page delivered (ntfy) FIRING OrdersApiAvailabilityFastBurn
   10:02:51Z  [you]     action     - ack; declaring SEV1 (customer-facing 5xx); roles: IC+investigator+scribe = me
   10:03:10Z  [you]     action     - status update #1 posted: investigating elevated errors on orders-api, next update 10:30Z

   Post the first status update within the first few minutes even though you know nothing yet; the template makes that a thirty-second task.
3. 3.3
   Investigate in the runbook's order, and mitigate the moment you have a plausible lever. Log each observation as a line. The dashboard's status-code panel, the app logs and git log on deploy/ answer most scenarios; a stopped Prometheus (scenario B) shows up as a flat dashboard and an Alertmanager with no data — notice it and restart it before anything else.

   bashCopy

   docker compose -f deploy/compose.yml ps
   docker compose -f deploy/compose.yml logs app --tail 100 | grep -v ' 200 '
   git -C deploy log --oneline -5 -- .
   grep CHAOS deploy/.env
   docker compose -f deploy/compose.yml exec app sh -c 'tc qdisc show dev eth0 2>/dev/null || echo no-netem'

   Check: The timeline shows an observation naming the cause (or a decision to mitigate blindly by rollback while investigation continues) within ten minutes of the page.
4. 3.4
   Mitigate with the runbook command, confirm on the dashboard, post update #2, and keep watching until the SLI has been within target for fifteen minutes. Then resolve: post the resolution, the impact in plain terms (minutes of burn, requests failed from the Prometheus counter), and name the postmortem owner.

   promqlCopy

   # requests failed during the incident window (put the times in)
   sum(increase(http_requests_total{job="app", status=~"5.."}[25m] offset 0m))
   # error budget consumed by the incident, as a fraction of the 30-day budget
   (sum(increase(http_requests_total{job="app", status=~"5.."}[25m])) / sum(increase(http_requests_total{job="app"}[25m]))) / 0.001 * (25 / (30*24*60))

   Check: The timeline ends with a resolved line, and the three numbers are computed: time to acknowledge (page → ack), time to mitigate (page → SLI recovering), time to resolve (page → resolved).
5. 3.5
   Immediately after resolving, while it is fresh, write three bullet points in the timeline file under "hot notes": what was confusing, what took longest, and what you wished you had. These become the seed of the postmortem and are lost by tomorrow if not written now.

   Check: The timeline file has a hot-notes section with at least three bullets and the final numbers.

A blameless postmortem with action items, one improvement actually shipped, and a second drill that shows the numbers moving.

1. 4.1
   Within two days, write the postmortem from the timeline using the template. The root cause section describes the system: not "the breaker set the error rate" but "an environment variable can put the service into a failing state with no deploy, no alert on the config change, and no guard". Contributing factors: what slowed detection or mitigation.

   Check: Every section filled; the detection section states the page latency and what would have caught it sooner; at least three action items with owners and dates, tagged prevent, detect, mitigate or process.
2. 4.2
   Hold a thirty-minute review with the breaker (they know what they did and what they expected you to do). Go through the timeline together and agree the action items. Typical outcomes from this drill: a config-change alert (any change to .env on the host emits an event), a RELEASES.md that the rollback command can actually use, a Grafana annotation from every deploy, a status-update reminder timer.

   If the breaker chose scenario C (broken alert rule), the headline action is a promtool test rules gate in CI and a "no page fired" detection: an alert on absent(sli:availability_error_ratio:rate5m) or on rule evaluation failures.
3. 4.3
   Ship one action item before the next drill, and make it testable. The config-change detection is a good first one: the host emits a log line whenever deploy/.env changes, and a Loki-less version just alerts on the file's modification time through a textfile collector metric.

   bashCopy

   # deploy/env-watch.sh: run by a cron every minute on the host (or a sidecar in compose)
   #!/bin/bash
   MTIME=$(stat -c %Y /opt/app/.env)
   mkdir -p /var/lib/node_exporter/textfile
   echo "app_env_file_mtime_seconds $MTIME" > /var/lib/node_exporter/textfile/app_env.prom
   
   # deploy/rules/config_alerts.yml
   # - alert: OrdersApiConfigChanged
   #   expr: changes(app_env_file_mtime_seconds[10m]) > 0
   #   labels: {severity: ticket, service: orders-api}
   #   annotations: {summary: "deploy/.env on the host changed (no deploy pipeline run)"}

   node-exporter reads *.prom files from a directory given by --collector.textfile.directory; mount it in Compose. The alert is a ticket, not a page, but during an incident it is the first thing the investigator sees on the alerts page.
4. 4.4
   Add a Grafana annotation to every deploy so the dashboard shows deploy markers; a page that starts at a marker points straight at a rollback. The release workflow posts to Grafana's annotations API after the SSM deploy command.

   bashCopy

   curl -s -X POST "http://$GRAFANA_HOST:3000/api/annotations" \
     -H "Authorization: Bearer $GRAFANA_TOKEN" -H 'Content-Type: application/json' \
     -d "{\"tags\":[\"deploy\",\"orders-api\"],\"text\":\"deploy ${IMAGE_DIGEST:7:12} by ${GITHUB_ACTOR}\"}"

   Create a Grafana service account token with the Editor role and store it as a repository secret; in the local lab, Grafana is only reachable from your IP, so run the same curl from the deploy runbook instead of CI.
5. 4.5
   Run drill two, a week later if you can, with a different scenario and ideally a different breaker. Same process, same timeline discipline. Compare the three numbers with drill one, and check whether the shipped improvement changed the response (it should appear in the timeline as the moment the cause became obvious).

   Check: Drill two's timeline, postmortem and numbers are in incidents/, and the postmortem's "what went well" names the improvement from drill one.
6. 4.6
   Close out: a short incidents/README.md that links the process, the two drills and their numbers, and a table of action items with status. Merge it. You now have an on-call practice, not just an alert.

   bashCopy

   pkill hey || true
   git add incidents deploy chaos && git commit -m "incidents: process, roles, templates, two drills with postmortems, config-change detection"
   git push -u origin HEAD && gh pr create --fill && gh pr merge --squash --delete-branch

No push arrives although Alertmanager shows the alert

Check the relay's logs (docker compose logs relay); a wrong NTFY_TOPIC, a missing Content-Length (Alertmanager always sends it) or an outbound network block are the usual causes. curl -d test https://ntfy.sh/TOPIC from inside the relay container isolates the network.

The page arrives, but the RESOLVED notification never does

send_resolved: true must be set on the receiver, and the alert must actually resolve in Prometheus (both windows of the burn-rate expression below threshold). With the 1h window it can take a while after the fix; the SLO project explains why.

The escalation route fires immediately instead of after 15 minutes

group_wait applies per route group; make sure the escalation route has its own group_wait: 15m and that continue: true is on the primary route so the alert reaches both. Test with amtool config routes test --config.file alertmanager.yml severity=page.

The breaker's scenario took the whole stack down and the drill cannot proceed

That is a finding: record it, restore with the chaos/README.md command, and add "blast radius of the breaker's tools" to the rules. In production the equivalent is a game day with a pre-agreed abort criterion.

The postmortem turns into a discussion of who did what

Reframe every sentence about a person as a sentence about the system: 'the variable could be changed without a record' rather than 'X changed the variable'. If the review cannot do that, stop and restart it with the blameless section of the SRE track's postmortem module read aloud.

The timeline is empty because everyone was busy fixing

That is the most common failure and the reason the scribe role exists. Solo responders: write one line per action before executing it; it costs five seconds and is the only record you will have.

• Replace ntfy with a pager product's free tier to get real acknowledgement, escalation policies and on-call schedules, and run drill three on a schedule handover.
• Run the drill against the GitOps cluster from the DevOps track: mitigation becomes git revert, and the config-change detection becomes Argo CD's OutOfSync notification.
• Add a customer-facing status page (a static page updated from the status-update template) and include publishing it in the comms role.
• Take the Cybersecurity track's incident response module and run a security drill with the same process; the roles and timeline are identical, the containment steps differ.
• Compute error budget consumed per incident automatically from Prometheus and put it in the postmortem template's impact section.