SLIs, SLOs & error budgets

SLI, SLO, SLA

SLIs are what you measure, SLOs are what you aim for, and SLAs are what you promise. An SLA is a business and legal document, and most internal services have none. Where one exists, set the internal SLO stricter than the SLA, so that you get a warning, and time to react, well before you owe anybody money.

The subtle point is that an SLO is both a floor and a ceiling. Falling below it means users are unhappy and reliability work takes priority. But greatly exceeding it also carries a cost: you could have shipped faster or spent less, and users come to depend on a level of service you never promised. Google has deliberately taken down an over-reliable internal service for short periods so that its consumers could not quietly build a dependency on it never failing.

Choosing good SLIs

A good SLI rises and falls with user happiness. CPU utilisation is not an SLI: users do not care about it, and it can be at ninety percent while everyone is perfectly happy. "Did my page load, quickly and correctly?" is what they care about. The Google SRE Workbook recommends expressing every SLI the same way, as a ratio:

SLI = good events / valid events x 100%

valid events : everything that should count   (e.g. all HTTP requests except health checks)
good events  : those that met the criterion   (e.g. status below 500 AND served in < 500 ms)

That form has real advantages. It always runs from 0 to 100%, where 100% is perfect. It is easy to reason about, the error budget drops straight out of it, and the same alerting logic works for every SLI you define. Which indicators to use depends on the kind of system.

• Measure as close to the user as you can. Server-side metrics are easy but miss failures in the load balancer, DNS or the network. Load balancer logs see more; synthetic probes and real-user monitoring see the most. Start with what you have and move outward.
• Define latency as a proportion under a threshold, not as an average. Averages hide the slow tail, and "99% of requests under 500 ms" fits the good-over-valid form. Use two thresholds if needed, for example 90% under 200 ms and 99% under 1 s.
• Decide what is valid. Exclude health checks and traffic from your own load tests. Think carefully about 4xx responses: a 404 for a page that does not exist is the service working correctly, while a wave of 401 errors caused by a broken auth deploy is not.
• Keep the number small. Two to four SLIs per user journey, for the journeys that matter most: log in, search, check out. Not one for every endpoint.

# availability: proportion of requests that did not fail server-side
sum(rate(http_requests_total{job="checkout", code!~"5.."}[28d]))
/
sum(rate(http_requests_total{job="checkout"}[28d]))

# latency: proportion of requests served in under 500 ms (histogram bucket le="0.5")
sum(rate(http_request_duration_seconds_bucket{job="checkout", le="0.5"}[28d]))
/
sum(rate(http_request_duration_seconds_count{job="checkout"}[28d]))

Setting the target

Do not start from what sounds impressive. Start from two questions: what do users need, and what does the system do today?

1. Measure current performance over the last month or quarter. If the service has delivered 99.7%, an SLO of 99.99% is fiction that will be in breach from its first day and ignored by the second week.
2. Set the first target at or slightly below what you already achieve, as long as users are reasonably happy at that level. An achievable SLO that people act on is worth more than an ambitious one nobody believes.
3. Check it against your dependencies. A service cannot be more available than the critical things it depends on, unless you add redundancy, caching or graceful degradation. If every request needs a database with a 99.95% objective, promising 99.99% is not credible.
4. Choose a window. A rolling window of 28 or 30 days is the common choice. Four weeks always contains the same number of weekends, which makes periods comparable. Calendar months suit reporting to the business but reset abruptly.
5. Agree it with the product owner, in writing. The SLO is a product decision about how reliable the feature needs to be. Engineering cannot set it alone.
6. Review it every quarter or so. Tighten it if users complain while you are meeting it, and loosen it if you keep missing it while nobody notices.

apiVersion: openslo/v1
kind: SLO
metadata:
  name: checkout-availability
  displayName: Checkout requests succeed
spec:
  service: checkout
  description: Proportion of checkout API requests that do not fail with a 5xx
  budgetingMethod: Occurrences
  timeWindow:
    - duration: 28d
      isRolling: true
  objectives:
    - displayName: Successful requests
      target: 0.999

Error budgets

If the target is 99.9%, then 0.1% of events are allowed to fail. That allowance is the error budget. It is not a number to minimise. It is a resource to spend, on releases, migrations, experiments, chaos tests and planned maintenance, all of which carry some risk of failure.

error budget (fraction)  = 1 - SLO                         99.9%  ->  0.1%
error budget (events)    = (1 - SLO) x valid events        10,000,000 requests  ->  10,000 may fail
error budget (time)      = (1 - SLO) x window              28 days  ->  40.3 minutes of full outage

budget consumed          = bad events so far / error budget in events
budget remaining         = 1 - budget consumed

Work an example. A service receives 10 million requests in 28 days with a 99.9% SLO, so the budget is 10,000 failed requests. A bad deploy causes 2,500 errors, a dependency outage another 1,500. That is 4,000 bad events, so 40% of the budget is consumed and 60% remains. An incident in which 20% of requests fail for two hours costs far less than a total outage of the same length, and a request-based budget accounts for that correctly, which is why it is better than counting minutes of downtime.

Burn rate tells you how fast the budget is being spent, relative to the rate that would use it up exactly at the end of the window.

burn rate = observed error rate / (1 - SLO)

SLO 99.9%  ->  allowed error rate 0.1%
  error rate 0.1%   ->  burn rate 1     budget lasts exactly the window (28 days)
  error rate 0.2%   ->  burn rate 2     budget gone in 14 days
  error rate 1.44%  ->  burn rate 14.4  budget gone in under 2 days
  error rate 100%   ->  burn rate 1000  budget gone in about 40 minutes

time to exhaustion = window / burn rate

def error_budget_report(slo: float, valid: int, bad: int, window_days: int = 28) -> dict:
    """slo as a fraction, e.g. 0.999. valid/bad are event counts so far in the window."""
    budget_events = (1 - slo) * valid
    consumed = bad / budget_events
    return {
        "sli": 1 - bad / valid,
        "budget_events": round(budget_events),
        "budget_consumed": round(consumed, 3),
        "budget_remaining": round(1 - consumed, 3),
        "full_outage_minutes_allowed": round((1 - slo) * window_days * 24 * 60, 1),
    }


print(error_budget_report(slo=0.999, valid=10_000_000, bad=4_000))
# sli 0.9996, budget 10000 events, 0.4 consumed, 0.6 remaining, 40.3 minutes

Burn rate is the basis of good alerting, which the alerting module covers: a high burn rate for a short time deserves a page, and a modest burn rate sustained for days deserves a ticket.

The error budget policy

An SLO that changes nobody's behaviour is just a dashboard. The error budget policy is a short document, agreed in advance by engineering, product and management, that says what happens as the budget is spent. Agreeing it before the crisis is the point: during an incident nobody negotiates well.

Service: checkout          SLO: 99.9% availability, rolling 28 days
Owners: checkout team (engineering), J. Rivera (product)

Budget remaining > 50%
  Normal operation. Releases proceed at will. Risky work (migrations,
  chaos experiments) is encouraged while there is budget to absorb it.

Budget remaining 0-50%
  Releases continue. Reliability items from recent postmortems move to
  the top of the next sprint. Risky work needs the on-call's agreement.

Budget exhausted (SLO missed over the window)
  Feature releases stop. Only reliability fixes, security patches and
  changes required by law ship until the service is back within SLO.
  The team spends its time on the causes identified in postmortems.

A single incident consuming > 20% of the budget
  A postmortem is mandatory, with at least one priority action item.

Exceptions and disputes
  Escalate to the engineering director, who decides.
  Outages caused by another team's service are attributed to that service.

Notice what the policy does to incentives. Developers want to ship, and now the way to keep shipping is to keep the service reliable, so they start asking for canaries and better tests themselves. Operators who wanted to block risky changes have no grounds to do so while budget remains. The argument about whether it is safe to release has been replaced by a number both sides accepted beforehand.