SLOs from scratch

The service, Prometheus, Alertmanager and Grafana running locally with steady traffic flowing, so every later rule has data to evaluate.

1. 1.1
   Clone the zero-to-production repository (or your own service) and create a branch for the SLO work.

   bashCopy

   gh repo clone YOUR_GITHUB_USER/zero-to-prod && cd zero-to-prod
   git switch -c feat/slos

2. 1.2
   Add an endpoint that does real-looking work and can be made to fail on demand. The failure rate comes from an environment variable so you can inject errors without redeploying code.

   pythonCopy

   # app/main.py (add to the existing FastAPI app)
   import os
   import random
   import time
   
   from fastapi import HTTPException
   
   
   @app.get("/orders/{order_id}")
   def get_order(order_id: int):
       time.sleep(random.uniform(0.02, 0.12))            # simulated work: 20-120 ms
       if random.random() < float(os.environ.get("CHAOS_ERROR_RATE", "0")):
           raise HTTPException(status_code=503, detail="upstream unavailable")
       if random.random() < float(os.environ.get("CHAOS_SLOW_RATE", "0")):
           time.sleep(0.6)                                # occasional slow response
       return {"order_id": order_id, "status": "paid"}

   Keep the existing /, /health and /metrics endpoints. The instrumentator already records http_requests_total{handler,method,status} and the http_request_duration_seconds histogram for every route.
3. 1.3
   Add Alertmanager to the Compose stack and pass the chaos variables through to the app, defaulting to zero.

   yamlCopy

   # deploy/compose.yml (additions)
   services:
     app:
       environment:
         GREETING: ${GREETING:-hello from compose}
         CHAOS_ERROR_RATE: ${CHAOS_ERROR_RATE:-0}
         CHAOS_SLOW_RATE: ${CHAOS_SLOW_RATE:-0}
   
     alertmanager:
       image: prom/alertmanager:v0.28.0
       volumes:
         - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
       ports: ["9093:9093"]
       restart: unless-stopped
   
     prometheus:
       volumes:
         - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
         - ./rules:/etc/prometheus/rules:ro
         - prom-data:/prometheus
       command:
         - --config.file=/etc/prometheus/prometheus.yml
         - --storage.tsdb.retention.time=15d

   Merge these into the existing service definitions rather than duplicating them; the prometheus volumes list replaces the old one that mounted alerts.yml directly.
4. 1.4
   Point Prometheus at the rules directory and at Alertmanager, and lower the scrape interval so short drills produce enough samples.

   yamlCopy

   # deploy/prometheus.yml
   global:
     scrape_interval: 10s
     evaluation_interval: 10s
   
   rule_files:
     - /etc/prometheus/rules/*.yml
   
   alerting:
     alertmanagers:
       - static_configs:
           - targets: ["alertmanager:9093"]
   
   scrape_configs:
     - job_name: app
       static_configs:
         - targets: ["app:8000"]
     - job_name: node
       static_configs:
         - targets: ["node-exporter:9100"]
     - job_name: prometheus
       static_configs:
         - targets: ["localhost:9090"]

5. 1.5
   Give Alertmanager a minimal configuration for now: everything goes to a webhook receiver that logs to a tiny sink container, so you can see alerts arrive without needing Slack or a pager yet.

   yamlCopy

   # deploy/alertmanager.yml
   route:
     receiver: default
     group_by: [alertname, service]
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 4h
   
   receivers:
     - name: default
       webhook_configs:
         - url: http://alert-sink:8080/
           send_resolved: true

   Stop here and add the sink service in the next step; the URL above points at it by service name.
6. 1.6
   Add the alert sink container to the Compose stack. It is an HTTP echo server that prints every request it receives, so alert deliveries show up in its logs.

   yamlCopy

   # deploy/compose.yml (addition)
   services:
     alert-sink:
       image: mendhak/http-https-echo:35
       environment:
         HTTP_PORT: "8080"
       restart: unless-stopped

   Check: docker compose -f deploy/compose.yml config prints the merged file without errors and lists alert-sink among the services.
7. 1.7
   Rebuild and start the stack, then generate steady traffic against the new endpoint. Install hey (brew install hey on macOS; a binary release on Linux) or use the shell loop.

   bashCopy

   docker compose -f deploy/compose.yml build app
   docker compose -f deploy/compose.yml up -d
   
   # 20 requests/second for 10 minutes, in the background
   hey -z 10m -q 20 -c 2 http://localhost/orders/42 > /dev/null 2>&1 &
   # without hey:
   # while true; do curl -s -o /dev/null http://localhost/orders/42; sleep 0.05; done &

   Check: http://localhost:9090/targets shows app, node and prometheus as UP. In the Prometheus graph, sum(rate(http_requests_total{handler="/orders/{order_id}"}[1m])) is about 20.

Two SLIs expressed as Prometheus expressions, one SLO document that a product manager could sign, and the error budget that follows from it.

1. 2.1
   Decide the SLIs as ratios of good events to valid events, using only the metrics the service already exports. Availability: requests that did not return a 5xx. Latency: requests served under 250 ms (the instrumentator's histogram has a 0.25 bucket). Health checks and metrics scrapes are excluded because they are not user traffic.

   textCopy

   availability SLI = 1 - ( 5xx responses / all responses )      handler != /health, /metrics
   latency SLI      = responses faster than 250 ms / all responses  same handlers
   
   valid events: every HTTP response to a user-facing handler
   good events:  status !~ 5.. (availability);  duration <= 0.25 s (latency)

2. 2.2
   Check both SLIs in the Prometheus UI as raw expressions before writing any rules. They should sit near 1.0 with no chaos injected.

   promqlCopy

   # availability over the last 5 minutes
   1 - (
     sum(rate(http_requests_total{job="app", handler!~"/health|/metrics", status=~"5.."}[5m]))
     /
     sum(rate(http_requests_total{job="app", handler!~"/health|/metrics"}[5m]))
   )
   
   # latency: fraction of requests under 250 ms over the last 5 minutes
   sum(rate(http_request_duration_seconds_bucket{job="app", handler!~"/health|/metrics", le="0.25"}[5m]))
   /
   sum(rate(http_request_duration_seconds_count{job="app", handler!~"/health|/metrics"}[5m])

   Check: Both expressions return a value close to 1 (for example 0.998 and 0.97). If the latency one is lower than you expect, the simulated work is doing its job; that is fine.
3. 2.3
   Write the SLO document. Pick targets the service can currently meet with a little headroom; you can tighten later. State the window, the SLIs, the budget, and who owns it.

   textCopy

   # slo/SLO.md
   
   ## orders-api service level objectives
   
   Owner: platform team (@you)          Window: rolling 30 days          Reviewed: quarterly
   
   | SLO          | SLI                                                   | Target  | Error budget / 30 d |
   |--------------|-------------------------------------------------------|---------|---------------------|
   | Availability | non-5xx responses / all responses (user handlers)     | 99.9 %  | 0.1 % of requests   |
   | Latency      | responses <= 250 ms / all responses (user handlers)   | 95.0 %  | 5 % of requests     |
   
   At 20 requests/s the 30-day availability budget is about 51,800 failed requests,
   or 43 minutes of total outage.
   
   ### Error budget policy
   - Budget remaining > 50 %: ship normally.
   - Budget remaining 10-50 %: releases need an SRE reviewer; no risky migrations.
   - Budget exhausted: feature releases freeze until the budget recovers; only reliability
     work and fixes ship. The freeze is lifted when the 30-day SLI is back above target.
   - A fast-burn page (see alerts) triggers an incident regardless of remaining budget.
   
   ### Exclusions
   /health and /metrics are not user traffic and are excluded from both SLIs.

   The arithmetic: 20 req/s × 86,400 s × 30 d ≈ 51.8 million requests; 0.1 % of that is about 51,800; at full outage (20 failures/s) that budget lasts 2,590 s ≈ 43 minutes.

The SLIs, error ratios and burn rates as recorded series, and multi-window multi-burn-rate alerts that page for fast burns and ticket for slow ones.

1. 3.1
   Create the recording rules. Each SLI's error ratio is recorded over several windows, because burn-rate alerts compare a short and a long window. Naming follows the level:metric:operations convention.

   yamlCopy

   # deploy/rules/slo_recording.yml
   groups:
     - name: slo_recording
       interval: 10s
       rules:
         # --- availability error ratio (bad / total) over several windows ---
         - record: sli:availability_error_ratio:rate5m
           expr: |
             sum(rate(http_requests_total{job="app", handler!~"/health|/metrics", status=~"5.."}[5m]))
             / sum(rate(http_requests_total{job="app", handler!~"/health|/metrics"}[5m]))
         - record: sli:availability_error_ratio:rate30m
           expr: |
             sum(rate(http_requests_total{job="app", handler!~"/health|/metrics", status=~"5.."}[30m]))
             / sum(rate(http_requests_total{job="app", handler!~"/health|/metrics"}[30m]))
         - record: sli:availability_error_ratio:rate1h
           expr: |
             sum(rate(http_requests_total{job="app", handler!~"/health|/metrics", status=~"5.."}[1h]))
             / sum(rate(http_requests_total{job="app", handler!~"/health|/metrics"}[1h]))
         - record: sli:availability_error_ratio:rate6h
           expr: |
             sum(rate(http_requests_total{job="app", handler!~"/health|/metrics", status=~"5.."}[6h]))
             / sum(rate(http_requests_total{job="app", handler!~"/health|/metrics"}[6h]))
         - record: sli:availability_error_ratio:rate3d
           expr: |
             sum(rate(http_requests_total{job="app", handler!~"/health|/metrics", status=~"5.."}[3d]))
             / sum(rate(http_requests_total{job="app", handler!~"/health|/metrics"}[3d]))
   
         # --- latency error ratio (slow / total) ---
         - record: sli:latency_error_ratio:rate5m
           expr: |
             1 - (
               sum(rate(http_request_duration_seconds_bucket{job="app", handler!~"/health|/metrics", le="0.25"}[5m]))
               / sum(rate(http_request_duration_seconds_count{job="app", handler!~"/health|/metrics"}[5m]))
             )
         - record: sli:latency_error_ratio:rate1h
           expr: |
             1 - (
               sum(rate(http_request_duration_seconds_bucket{job="app", handler!~"/health|/metrics", le="0.25"}[1h]))
               / sum(rate(http_request_duration_seconds_count{job="app", handler!~"/health|/metrics"}[1h]))
             )
         - record: sli:latency_error_ratio:rate6h
           expr: |
             1 - (
               sum(rate(http_request_duration_seconds_bucket{job="app", handler!~"/health|/metrics", le="0.25"}[6h]))
               / sum(rate(http_request_duration_seconds_count{job="app", handler!~"/health|/metrics"}[6h]))
             )
   
         # --- 30-day SLI and remaining budget, for the dashboard ---
         - record: slo:availability_sli:ratio_rate30d
           expr: |
             1 - (
               sum(rate(http_requests_total{job="app", handler!~"/health|/metrics", status=~"5.."}[30d]))
               / sum(rate(http_requests_total{job="app", handler!~"/health|/metrics"}[30d]))
             )
         - record: slo:availability_error_budget_remaining:ratio
           expr: |
             1 - ((1 - slo:availability_sli:ratio_rate30d) / (1 - 0.999))

   A 15-day retention (set in Compose) means the 30-day series will be based on partial data until the stack has run that long; that is fine for the drill.
2. 3.2
   Write the alerts. A burn rate of 1 consumes exactly the budget over the window; 14.4 consumes the 30-day budget in two days. The pairs of windows (long and short) stop an alert from lingering after the problem is fixed. These thresholds are the standard multi-window, multi-burn-rate recommendation for a 30-day SLO.

   yamlCopy

   # deploy/rules/slo_alerts.yml
   groups:
     - name: slo_alerts
       rules:
         # 99.9 % availability -> error budget 0.001
         - alert: OrdersApiAvailabilityFastBurn
           expr: |
             sli:availability_error_ratio:rate1h > (14.4 * 0.001)
             and sli:availability_error_ratio:rate5m > (14.4 * 0.001)
           for: 2m
           labels: {severity: page, service: orders-api, slo: availability}
           annotations:
             summary: "orders-api is burning its 30-day availability budget in ~2 days"
             runbook: "https://github.com/YOUR_GITHUB_USER/zero-to-prod/blob/main/slo/RUNBOOK-orders-api.md"
   
         - alert: OrdersApiAvailabilitySlowBurn
           expr: |
             sli:availability_error_ratio:rate6h > (6 * 0.001)
             and sli:availability_error_ratio:rate30m > (6 * 0.001)
           for: 15m
           labels: {severity: page, service: orders-api, slo: availability}
           annotations:
             summary: "orders-api is burning its availability budget in ~5 days"
   
         - alert: OrdersApiAvailabilityBudgetTrend
           expr: |
             sli:availability_error_ratio:rate3d > (1 * 0.001)
             and sli:availability_error_ratio:rate6h > (1 * 0.001)
           for: 1h
           labels: {severity: ticket, service: orders-api, slo: availability}
           annotations:
             summary: "orders-api availability is on track to exhaust its budget this month"
   
         # 95 % latency -> error budget 0.05
         - alert: OrdersApiLatencyFastBurn
           expr: |
             sli:latency_error_ratio:rate1h > (14.4 * 0.05)
             and sli:latency_error_ratio:rate5m > (14.4 * 0.05)
           for: 2m
           labels: {severity: page, service: orders-api, slo: latency}
           annotations:
             summary: "orders-api latency SLO burning fast: >72% of requests slower than 250 ms"
   
         - alert: OrdersApiLatencySlowBurn
           expr: |
             sli:latency_error_ratio:rate6h > (6 * 0.05)
             and sli:latency_error_ratio:rate1h > (6 * 0.05)
           for: 15m
           labels: {severity: ticket, service: orders-api, slo: latency}
           annotations:
             summary: "orders-api latency SLO burning: >30% of requests slower than 250 ms for 6 h"

   14.4 × 0.05 = 0.72, so the latency fast-burn only fires when most requests are slow — appropriate for a 95 % target. For a tighter latency SLO the thresholds scale down automatically.
3. 3.3
   Validate the rule files with promtool before reloading Prometheus.

   bashCopy

   docker run --rm -v "$PWD/deploy/rules:/rules:ro" --entrypoint promtool prom/prometheus:v3.1.0 check rules /rules/slo_recording.yml /rules/slo_alerts.yml

   Check: Both files report SUCCESS with the number of rules found.
4. 3.4
   Unit-test the fast-burn alert with synthetic series, so the arithmetic is proven before any traffic exists. The test feeds a 2 % error ratio for both windows and expects the page after the for duration.

   yamlCopy

   # deploy/tests/slo_alerts_test.yml
   rule_files:
     - ../rules/slo_alerts.yml
   
   evaluation_interval: 1m
   
   tests:
     - interval: 1m
       input_series:
         - series: 'sli:availability_error_ratio:rate1h'
           values: '0.02x10'
         - series: 'sli:availability_error_ratio:rate5m'
           values: '0.02x10'
       alert_rule_test:
         - eval_time: 5m
           alertname: OrdersApiAvailabilityFastBurn
           exp_alerts:
             - exp_labels:
                 severity: page
                 service: orders-api
                 slo: availability
               exp_annotations:
                 summary: "orders-api is burning its 30-day availability budget in ~2 days"
                 runbook: "https://github.com/YOUR_GITHUB_USER/zero-to-prod/blob/main/slo/RUNBOOK-orders-api.md"
         - eval_time: 1m
           alertname: OrdersApiAvailabilityFastBurn
           exp_alerts: []          # not yet: the 'for: 2m' has not elapsed

5. 3.5
   Run the unit test, then reload Prometheus so the rules are live.

   bashCopy

   docker run --rm -v "$PWD/deploy:/deploy:ro" --entrypoint promtool prom/prometheus:v3.1.0 test rules /deploy/tests/slo_alerts_test.yml
   docker compose -f deploy/compose.yml restart prometheus

   Check: promtool test rules prints SUCCESS. http://localhost:9090/rules lists both groups with no errors, and http://localhost:9090/alerts shows the five alerts as inactive (green).

Pages and tickets go to different places, an SLO dashboard shows the budget at a glance, and both are provisioned from the repository.

1. 4.1
   Route by severity in Alertmanager: page goes to the on-call receiver, ticket to a low-urgency receiver. Both are webhooks to the sink for now; the receiver blocks are where a real Slack, PagerDuty or Opsgenie integration goes later.

   yamlCopy

   # deploy/alertmanager.yml
   route:
     receiver: tickets
     group_by: [alertname, service]
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 4h
     routes:
       - matchers: [severity="page"]
         receiver: oncall
         group_wait: 10s
         repeat_interval: 1h
       - matchers: [severity="ticket"]
         receiver: tickets
   
   inhibit_rules:
     # while a fast burn is paging, do not also send the slow-burn or trend alerts for the same SLO
     - source_matchers: [alertname=~".*FastBurn"]
       target_matchers: [alertname=~".*(SlowBurn|BudgetTrend)"]
       equal: [service, slo]
   
   receivers:
     - name: oncall
       webhook_configs:
         - url: http://alert-sink:8080/oncall
           send_resolved: true
     - name: tickets
       webhook_configs:
         - url: http://alert-sink:8080/tickets
           send_resolved: true

   Validate with docker run --rm -v "$PWD/deploy/alertmanager.yml:/a.yml:ro" --entrypoint amtool prom/alertmanager:v0.28.0 check-config /a.yml, then docker compose -f deploy/compose.yml restart alertmanager.
2. 4.2
   Replace the dashboard JSON with an SLO view: the 30-day SLI against its target, budget remaining, the burn rates for each window, and the raw error ratios. Grafana's provisioning from the zero-to-production project already loads any JSON in the dashboards folder.

   jsonCopy

   {
     "title": "orders-api SLOs",
     "uid": "orders-api-slo",
     "schemaVersion": 39,
     "refresh": "30s",
     "time": {"from": "now-6h", "to": "now"},
     "panels": [
       {"type": "stat", "title": "Availability SLI (30d) vs 99.9%", "gridPos": {"x": 0, "y": 0, "w": 8, "h": 6},
        "fieldConfig": {"defaults": {"unit": "percentunit", "decimals": 3,
          "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 0.999}]}}},
        "targets": [{"expr": "slo:availability_sli:ratio_rate30d"}]},
       {"type": "gauge", "title": "Availability error budget remaining", "gridPos": {"x": 8, "y": 0, "w": 8, "h": 6},
        "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "max": 1,
          "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "orange", "value": 0.1}, {"color": "green", "value": 0.5}]}}},
        "targets": [{"expr": "clamp_min(slo:availability_error_budget_remaining:ratio, 0)"}]},
       {"type": "stat", "title": "Latency SLI (1h) vs 95%", "gridPos": {"x": 16, "y": 0, "w": 8, "h": 6},
        "fieldConfig": {"defaults": {"unit": "percentunit", "decimals": 2,
          "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 0.95}]}}},
        "targets": [{"expr": "1 - sli:latency_error_ratio:rate1h"}]},
       {"type": "timeseries", "title": "Availability burn rate by window (1 = budget consumed exactly over 30d)", "gridPos": {"x": 0, "y": 6, "w": 24, "h": 9},
        "fieldConfig": {"defaults": {"custom": {"thresholdsStyle": {"mode": "line"}},
          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 6}, {"color": "red", "value": 14.4}]}}},
        "targets": [
          {"expr": "sli:availability_error_ratio:rate5m / 0.001", "legendFormat": "5m"},
          {"expr": "sli:availability_error_ratio:rate1h / 0.001", "legendFormat": "1h"},
          {"expr": "sli:availability_error_ratio:rate6h / 0.001", "legendFormat": "6h"}
        ]},
       {"type": "timeseries", "title": "Requests/s by status", "gridPos": {"x": 0, "y": 15, "w": 12, "h": 8},
        "targets": [{"expr": "sum by (status) (rate(http_requests_total{job=\"app\", handler!~\"/health|/metrics\"}[1m]))", "legendFormat": "{{status}}"}]},
       {"type": "timeseries", "title": "Latency p50 / p95 / p99", "gridPos": {"x": 12, "y": 15, "w": 12, "h": 8},
        "fieldConfig": {"defaults": {"unit": "s"}},
        "targets": [
          {"expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{job=\"app\", handler!~\"/health|/metrics\"}[5m])) by (le))", "legendFormat": "p50"},
          {"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"app\", handler!~\"/health|/metrics\"}[5m])) by (le))", "legendFormat": "p95"},
          {"expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job=\"app\", handler!~\"/health|/metrics\"}[5m])) by (le))", "legendFormat": "p99"}
        ]}
     ]
   }

   Save as deploy/grafana/dashboards/slo.json and restart Grafana (docker compose -f deploy/compose.yml restart grafana).
3. 4.3
   Open Grafana at http://localhost:3000 and find the orders-api SLOs dashboard. With clean traffic, the availability stat is green, budget remaining is near 100 %, and the burn-rate lines sit near zero.

   Check: All three stat panels have values; the burn-rate chart shows the 14.4 and 6 threshold lines.

Prove the whole chain — injected failure, SLI degradation, burn-rate alert, Alertmanager routing, dashboard — and record what you saw.

1. 5.1
   Write the runbook first, because the page will link to it. Keep it to what an on-call engineer needs in the first five minutes.

   textCopy

   # slo/RUNBOOK-orders-api.md
   
   ## OrdersApiAvailabilityFastBurn / SlowBurn
   
   **Meaning:** more than 1.4 % (fast) or 0.6 % (slow) of user requests are failing with 5xx.
   At this rate the 30-day error budget is gone in ~2 days (fast) or ~5 days (slow).
   
   **Confirm (2 min)**
   1. Dashboard: http://localhost:3000/d/orders-api-slo  - which status codes? since when?
   2. `docker compose -f deploy/compose.yml ps` - is the app container Up?
   3. `docker compose -f deploy/compose.yml logs app --tail 100` - what is the error?
   
   **Common causes**
   - Bad deploy: check the last release time against the start of the burn. Roll back:
     `docker compose -f deploy/compose.yml up -d app` with the previous APP_IMAGE in .env.
   - Upstream dependency down (503s): check the dependency's status; enable the fallback if one exists.
   - Chaos left on: `grep CHAOS deploy/.env` - CHAOS_ERROR_RATE must be 0 in production.
   
   **Mitigate first, then diagnose.** Rolling back or removing bad config stops the burn; the
   investigation continues afterwards in the incident channel.
   
   **Escalate** to the service owner if the burn continues 15 minutes after mitigation.
   
   ## OrdersApiLatencyFastBurn
   Same procedure; look at p95/p99 and the slow-rate chaos variable; check CPU on the host
   (node dashboard) and upstream latency.

2. 5.2
   Start the drill: keep the load generator running, then inject a 3 % error rate. That is 30 times the availability error budget, well above the fast-burn threshold of 1.44 %. Note the time.

   bashCopy

   date -u
   CHAOS_ERROR_RATE=0.03 docker compose -f deploy/compose.yml up -d app

   Compose recreates only the app container with the new environment; Prometheus keeps scraping through the restart.
3. 5.3
   Watch it propagate. Within about a minute the 5m error ratio rises above 0.0144; the 1h ratio follows more slowly because it averages over an hour of mostly good data — with 20 req/s and a fresh stack it crosses in a few minutes. Once both windows are above threshold for 2 minutes, the alert fires.

   Check: http://localhost:9090/alerts shows OrdersApiAvailabilityFastBurn as Pending, then Firing. http://localhost:9093 lists it under the oncall receiver. docker compose -f deploy/compose.yml logs alert-sink shows the webhook payload with the summary and the runbook URL.
4. 5.4
   Follow the runbook as if you were on call: open the dashboard, read the status breakdown (503s), check the container logs, find the chaos variable, and mitigate by removing it.

   bashCopy

   CHAOS_ERROR_RATE=0 docker compose -f deploy/compose.yml up -d app
   date -u

   Check: The 5m ratio drops immediately; the alert goes to inactive once the 5m window is clean (the 1h window may stay high longer, which is exactly why both windows are required). Alertmanager sends a resolved notification to the sink.
5. 5.5
   Repeat for latency: inject a 90 % slow rate (each slow response is 600 ms, well over the 250 ms threshold) and confirm OrdersApiLatencyFastBurn fires and resolves. Then run a slow-burn scenario: 0.8 % errors for 20 minutes should fire the slow-burn alert but not the fast one.

   bashCopy

   CHAOS_SLOW_RATE=0.9 docker compose -f deploy/compose.yml up -d app     # latency fast burn
   # ...observe, then reset...
   CHAOS_SLOW_RATE=0 CHAOS_ERROR_RATE=0.008 docker compose -f deploy/compose.yml up -d app   # slow burn only
   # ...wait ~20 min, observe, then reset...
   CHAOS_ERROR_RATE=0 docker compose -f deploy/compose.yml up -d app

   Check: The inhibition rule means that while a fast burn pages, the slow-burn alert for the same SLO is suppressed; in the slow-burn-only scenario it reaches the tickets receiver.
6. 5.6
   Record the drill in slo/DRILL-YYYY-MM-DD.md: the injection time, when the alert went Pending and Firing, when the webhook arrived, when you mitigated, when it resolved, and the time from injection to page. Then compute what the drill cost in error budget from the dashboard's budget-remaining gauge.

   Check: Time from injection to page is a few minutes; the budget gauge dropped by a visible amount and the runbook was sufficient to find the cause without reading this guide.
7. 5.7
   Stop the load generator, commit everything and merge through a pull request. Add a CI step that runs promtool check rules and promtool test rules so rule changes are validated before they reach the stack.

   bashCopy

   pkill hey || true
   git add app deploy slo
   git commit -m "feat: SLOs, burn-rate alerts, routing, dashboard, runbook and drill"
   git push -u origin feat/slos
   gh pr create --fill && gh pr merge --squash --delete-branch
   git switch main && git pull

   In ci.yml, add a job that runs the two promtool commands from this guide in a prom/prometheus container; a syntax error in a rule file would otherwise take down rule evaluation silently.

The SLI expressions return no data

Check the handler label values with count by (handler) (http_requests_total); FastAPI's instrumentator records the route template, e.g. /orders/{order_id}. Adjust the regex if your handlers differ, and make sure traffic is actually flowing (rate(...) over an empty window is empty).

promtool check rules fails with a parse error

Multi-line expressions must use the | block scalar and consistent indentation; a stray tab or a missing / at a line start is the usual cause. The error names the line.

The fast-burn alert never fires during the drill

The 1h window needs enough bad samples to cross 1.44 %. With a fresh stack and low traffic it may take longer; raise the injected rate to 0.1 for the drill or increase the load generator's rate. Check both recorded series in the Prometheus graph.

The alert fires but nothing reaches the sink

Check http://localhost:9093/#/status for the loaded config and http://localhost:9090/targets is not the right place — use http://localhost:9090/status for the Alertmanager discovery. Confirm the alert-sink container is Up and the receiver URL matches its port.

Grafana shows the old dashboard or none

Provisioned dashboards load from the folder on start; restart Grafana after adding the JSON, and check its logs for a JSON parse error (a trailing comma is the classic).

The 30-day SLI panel shows a value far from 1 after the drill

Expected: with only hours of data, a 30-day rate is dominated by the drill. It recovers as clean data accumulates. In production the window is real; in the lab, treat the 1h and 6h panels as the meaningful ones.

Latency SLI is below target even with no chaos

The simulated work (20-120 ms) plus container overhead may push some requests past 250 ms on a busy laptop. Either raise the threshold bucket to le="0.5" in the SLI, or lower the load generator's concurrency. Set the SLO to what the service can meet.

• Point the oncall receiver at a real pager (PagerDuty, Opsgenie or a Slack channel with escalation) and run the drill again with a colleague on call who does not know what you broke.
• Run the incident drill project from this track against the same stack: the page becomes a full incident with roles, a timeline and a blameless postmortem.
• Add the SLOs to a service catalogue entry and review the error budget in a weekly meeting for a month; adjust the targets from data.
• Move the rules and dashboard to Kubernetes with the Prometheus Operator (PrometheusRule and a dashboard ConfigMap) via the GitOps project in the DevOps track.
• Add an SLO for a dependency you call and see how its budget composes with yours.