Engineers tend to treat reliability as something to maximise. SRE treats it as something to choose. Every improvement in reliability is paid for with money, with engineering time, and with slower delivery of features, and beyond a certain point users cannot even notice what they are paying for. This module teaches you to reason about risk as a cost and a benefit, to decide how reliable a particular service needs to be, and to reduce the impact of failure in ways that are far cheaper than another nine.
- Explain why 100% reliability is the wrong target, in terms of cost, velocity and what users can perceive
- Estimate the cost of downtime and compare it with the cost of preventing it
- Calculate how dependencies combine, in series and in parallel, to limit a system's availability
- Set different risk tolerances for different services and user journeys
- Use risk reduction techniques that limit blast radius and recovery time instead of chasing failure prevention alone
Why not 100%?
Three arguments lead to the same conclusion.
Cost rises steeply. Moving from 99% to 99.9% might mean a second instance and a load balancer. Reaching 99.99% means multiple zones, automated failover, a rehearsed on-call rotation and careful releases. 99.999% means multiple regions, active-active data replication and removing humans from every recovery path. Each step costs several times the one before, for a tenth of the previous improvement.
Users cannot perceive it. Between your service and your user sit a phone, a Wi-Fi router, a mobile network, an ISP and DNS. If that chain is, generously, 99.9% reliable, a user cannot distinguish a 99.99% service from a 99.999% one. The failures they experience are dominated by everything else. The extra nine is real money spent on an improvement nobody can see.
Reliability competes with change. Most outages are caused by changes: deploys, config pushes, migrations. The surest way to be reliable is to change nothing, and a product that changes nothing loses to one that improves. A 100% target leaves an error budget of zero, which forbids every release.
The right target is the lowest level of reliability at which users remain happy, because every increment above it is paid for with features you did not build. Embracing risk does not mean being careless. It means choosing the level deliberately, measuring it, and managing to it.
Putting a price on downtime
To decide whether a reliability investment is worth making, compare what it costs with what it saves. The Google SRE book frames it as a simple calculation. Suppose a service earns 1,000,000 a year and you are considering a project to raise availability from 99.9% to 99.99%.
improvement in availability = 99.99% - 99.9% = 0.09%
value of the improvement = 1,000,000 x 0.0009 = 900 per year
If the project costs more than 900 per year, it is not justified by revenue alone.That model assumes revenue is lost in proportion to downtime, which is a simplification. A fuller estimate of the cost of an outage includes several components.
cost of an outage ~ lost revenue (revenue per hour x hours x share of users affected)
+ SLA penalties (credits owed to customers)
+ recovery cost (engineer hours x loaded hourly cost)
+ support cost (extra tickets and calls)
+ reputational damage (churn, lost deals: real, but hard to measure)
expected annual loss = probability of the event per year x cost when it happensdef expected_annual_loss(events_per_year: float, hours_each: float,
revenue_per_hour: float, share_affected: float,
responders: int, hourly_cost: float) -> float:
lost_revenue = hours_each * revenue_per_hour * share_affected
recovery = hours_each * responders * hourly_cost
return events_per_year * (lost_revenue + recovery)
# a zone failure: once every two years, 3 hours, all users, 4 responders
loss = expected_annual_loss(0.5, 3, 4_000, 1.0, 4, 120)
print(f"expected loss: {loss:,.0f} per year") # 6,720
# Running in a second zone costs 15,000 a year -> not justified by this risk alone.The numbers will be rough, and that is fine. The purpose is to make the trade-off visible and to have the conversation with the people who own the budget, in their terms. It also works in the other direction: for a payment system, the same arithmetic may show that a second region pays for itself many times over.
Dependencies set your ceiling
No service is more reliable than the things it cannot work without. When a request needs all of several components, their availabilities multiply, and the result is always worse than the weakest of them. This is a serial dependency.
serial (all must work): A_total = A1 x A2 x A3 x ...
app 99.95%, database 99.95%, auth service 99.9%, payment provider 99.9%
0.9995 x 0.9995 x 0.999 x 0.999 = 0.9970 -> 99.70%
expected downtime per 30 days: about 129 minutes, against 21.6 for the app by itself
redundant (any one is enough, failures independent):
A_total = 1 - (1 - A1) x (1 - A2) x ...
two instances at 99% each: 1 - 0.01 x 0.01 = 99.99%
three instances at 99% each: 1 - 0.01^3 = 99.9999%Two lessons follow. First, every hard dependency you add lowers your ceiling, so a microservice architecture with ten serial calls needs each service to be far more reliable than the target for the whole. A common rule of thumb is that a critical dependency should offer about one more nine than the service that depends on it.
Second, redundancy is powerful, but the formula assumes independent failures, and real ones often are not. Two replicas running the same bad release, reading the same corrupted config, in the same zone, behind the same load balancer, fail together. Redundancy pays off only to the extent that you remove the things the copies have in common.
The most effective way to escape the multiplication is to turn hard dependencies into soft ones. If the recommendation service is down, show the page without recommendations. If the cache is down, serve more slowly from the database. If a provider is slow, queue the work and confirm later. A dependency that can fail without failing the request drops out of the product entirely. This is graceful degradation, and the failure-modes module returns to it in depth.
Different services, different tolerances
There is no single correct target for an organisation. Decide per service, and often per user journey within a service, by asking a few questions.
- What do users expect? Compare with similar products and with what users currently tolerate. A free consumer tool and a paid enterprise API carry different expectations.
- Is revenue tied directly to availability? Checkout is; the recommendations carousel mostly is not.
- Which kind of failure matters? A brief period of errors, slow responses, stale data and lost data are very different. Losing data is usually far worse than being unavailable, and deserves a much stricter target.
- When does it matter? An internal reporting tool can be down at night at no cost. Planned maintenance windows are a legitimate way to spend the budget.
- What does it cost to serve each level, and who pays?
| Service | Reasonable target | Reasoning |
|---|---|---|
| Payment processing | 99.99%, with stricter durability | Direct revenue, trust, and no tolerance for lost transactions |
| Product search | 99.9% availability, latency SLO | Matters to revenue, but a retry is acceptable and degraded results are fine |
| Recommendations | 99% | A soft dependency: the page works without it |
| Internal analytics dashboard | 99% during business hours | A few users, and delay is an inconvenience, not a loss |
| Nightly batch export | Freshness: complete by 06:00 on 99% of days | Availability is the wrong measure; timeliness is what users feel |
Tiering has an architectural payoff. Once you accept that parts of the system can be less reliable, you can build them more cheaply: fewer replicas, spot instances, a single zone. You spend the savings where reliability is worth the most. Infrastructure teams can do the same by offering classes of service, for example a low-latency tier and a cheaper throughput tier, and letting consumers choose.
Reducing risk without buying another nine
Availability is driven by two things: how often you fail, and how long each failure lasts.
availability = MTBF / (MTBF + MTTR)
MTBF mean time between failures (how rarely it breaks)
MTTR mean time to restore (how quickly you recover)
fails every 30 days, 60 min to restore: 43,200 / 43,260 = 99.86%
fails every 30 days, 6 min to restore: 43,200 / 43,206 = 99.986%
fails every 300 days, 60 min to restore: 432,000 / 432,060 = 99.986%Recovering ten times faster buys the same availability as failing ten times less often, and it is nearly always cheaper and more achievable. Preventing all failures is impossible; recovering quickly is a matter of engineering. This is why so much of SRE concentrates on detection and recovery.
The third lever is blast radius: how many users a failure reaches. A request-based error budget counts the affected share, so a failure that touches 5% of users for an hour costs a twentieth of one that touches everyone.
| Lever | Techniques |
|---|---|
| Fail less often | Testing, code review, redundancy, removing single points of failure, simpler designs |
| Recover faster | Symptom-based alerting, one-command rollback, automated failover, runbooks, practised on-call |
| Affect fewer users | Canary releases, progressive rollout, feature flags, cell or shard isolation, regional independence |
| Fail more softly | Graceful degradation, timeouts, cached fallbacks, queueing instead of rejecting |
The error budget ties these together. While budget remains, take risks deliberately: release more often, run the migration, run the chaos experiment. Those are what the budget is for, and a team that never spends its budget is going slower than it needs to. When the budget is gone, stop taking discretionary risk until it recovers. That is embracing risk in practice: a quantity you manage, not a thing you fear.
A simple risk register helps you choose where to invest. List the plausible failure scenarios, estimate for each how often it happens, how long it lasts and what share of users it affects, and multiply to get expected bad minutes per year. Sort by that number and compare the top items with your error budget. The results regularly surprise teams, because frequent small failures often cost more than the rare dramatic one everybody worries about.