Someone has to answer when production breaks at three in the morning. How that duty is organised decides whether your team responds quickly and calmly for years, or burns out within months and starts making mistakes. On-call is a system to be engineered like any other: it has inputs you can measure, failure modes you can predict, and design choices that matter. This module covers rotations, escalation, handoffs and the practices that keep the humans in the loop healthy.
- Design a rotation with adequate size, shift length and primary and secondary cover
- Write an escalation policy with clear acknowledgement times and a defined path when nobody responds
- Run handoffs that transfer context, not just the pager
- Measure on-call load and keep it within sustainable limits
- Prepare new engineers for on-call through shadowing and practice, and protect responders from fatigue
What on-call is for
Being on call means being reachable and able to act within an agreed time, for an agreed period, in response to pages about production. The on-call engineer is the guardian of the service for that shift: they triage what comes in, mitigate what they can, and pull in help for what they cannot.
It is worth being clear about what the role is not. The on-call engineer is not expected to know everything, to fix root causes at night, or to work alone. Their goal during an incident is to restore service, usually by the quickest safe route: roll back, fail over, scale up, switch off a feature. Understanding why it happened is a job for daylight, with colleagues, in a postmortem.
Response-time expectations should follow from the SLO. If a service has a 99.99% target, the monthly budget is a little over four minutes, so a human response is already too slow and recovery must be automatic. At 99.9% there are about 43 minutes a month, which supports an expectation such as acknowledging within five minutes and being hands-on within fifteen. At 99% a response within the hour is reasonable, and the service may not need out-of-hours cover at all.
Not every service deserves a pager. Out-of-hours on-call is expensive in money and in people. If the business would not notice an outage until Monday, then alert during working hours only, and say so explicitly in the service's documentation.
Designing the rotation
A rotation needs enough people that each person's share is tolerable. The Google SRE book suggests at least eight engineers for a single-site rotation with week-long shifts, which puts each person on call roughly one week in eight and leaves room for holidays and illness. With two sites in different time zones, around six per site works, and it removes night shifts altogether: each site covers its own daytime. This arrangement is called follow the sun.
| Choice | Options | Trade-off |
|---|---|---|
| Shift length | One week is most common; some teams split weekdays and weekends | Longer shifts mean fewer handoffs but more accumulated fatigue |
| Cover | A primary, plus a secondary as backup | The secondary catches missed pages and provides a second pair of hands |
| Who participates | SREs only, or developers too | Developers on the rotation feel the operational cost of what they build |
| Hours | 24x7, extended hours, or business hours only | Match it to the SLO and to what the business really needs |
- Publish the schedule well ahead, at least a month, and make swaps easy and guilt-free. People have lives, and a rota that cannot bend will break.
- Never leave a gap. The paging tool should refuse a schedule with uncovered hours. Public holidays and the week a colleague leaves are where gaps appear.
- Small teams need honesty. A team of three cannot sustain 24x7 cover for long. Options are to share a rotation with a neighbouring team, reduce the hours covered, or invest in making the service need fewer pages.
- Compensate it. On-call is work. Time off in lieu, extra pay, or both, according to local law and custom. Uncompensated on-call breeds resentment, and then resignations.
Escalation policies
An escalation policy answers the question: what happens if the page is not acknowledged? Without one, a missed page, because of a flat battery, no signal, or deep sleep, becomes an outage that nobody is working on. The policy is configured in the paging tool and should be written down where everyone can read it.
service: checkout
severity_page:
- level: 1
notify: primary-oncall
methods: [push, sms, phone]
acknowledge_within: 5m
- level: 2
notify: secondary-oncall
methods: [push, sms, phone]
acknowledge_within: 10m
- level: 3
notify: engineering-manager
methods: [phone]
acknowledge_within: 15m
- level: 4
notify: director-of-engineering
methods: [phone]
repeat_policy: 2 # run through the levels twice before giving up
severity_ticket:
- notify: team-queue # no paging; handled next working dayThere are two kinds of escalation, and both must be easy. Hierarchical escalation moves up the chain when a page goes unanswered, as above. Functional escalation brings in someone with different expertise: the database team, the network team, the vendor. The on-call engineer must be able to reach those people quickly, which means other teams' on-call contacts are documented and their pagers can be triggered directly.
Make escalating a sign of good judgement, not of weakness. The rule to teach is: if you are not making progress after fifteen or twenty minutes, or if the impact is growing, bring someone in. Nobody has ever been criticised in a postmortem for asking for help too early.
Acknowledging a page means "I have seen this and I own it". It does not mean it is fixed. If you acknowledge and then find you cannot work on it, because you are driving, or unwell, hand it on explicitly by reassigning or escalating. A page that was acknowledged and then abandoned is worse than one never acknowledged, because escalation has been switched off.
Handoffs
A shift change transfers responsibility, and responsibility without context is dangerous. The incoming engineer needs to know what is fragile right now, what is in progress, and what they might be woken by. A short written note plus a ten-minute conversation is enough, and the written part matters because it survives if the conversation is skipped.
# On-call handoff: checkout (week 38 -> week 39)
Outgoing: Priya S. Incoming: Marco T. Secondary: Lena K.
## Open incidents / ongoing issues
- INC-2291 (mitigated, not resolved): payment provider timeouts. Retry budget
lowered as a workaround. Provider's fix expected Tuesday. Watch p99 latency.
## Pages this week: 4 (2 actionable, 1 duplicate, 1 false positive)
- 2x CheckoutErrorBudgetFastBurn: caused by INC-2291. Runbook was accurate.
- 1x QueueDepthHigh: duplicate of the above. Ticket OPS-812 to inhibit it.
- 1x CertExpirySoon: false positive, cert was already renewed. Ticket OPS-813.
## Things that may wake you
- Wednesday 02:00 UTC: database minor version upgrade (CHG-1042). Silence set.
- Marketing campaign Thursday 09:00: expect about double normal traffic.
## Temporary changes still in place (remember to undo)
- Retry budget for payment calls at 10% (normally 20%).
- Feature flag `new_address_form` is OFF in production.
## Runbook / tooling problems found
- Rollback runbook step 4 refers to the old cluster name. Fixed in PR 1187.The section on temporary changes earns its place. Workarounds applied during an incident, such as a raised limit, a disabled check or a manual override, are easily forgotten, and they become the hidden cause of the next incident. Write them down and give them an owner.
Review the week's pages at the handoff too. This is the natural moment for the alert review from the previous module: which pages were actionable, which should be deleted or demoted, and which point to toil that needs an engineering fix.
Keeping the load sustainable
Measure on-call load the same way you measure the service. What is not measured grows quietly until someone resigns.
| Metric | Why it matters | Healthy direction |
|---|---|---|
| Pages per shift | The basic measure of load | The SRE book suggests a maximum of about two incidents per 12-hour shift |
| Pages outside working hours | Sleep disruption drives fatigue and error | As close to zero as possible |
| Share of pages that were actionable | Noise erodes trust in the pager | Approaching 100% |
| Time to acknowledge | Shows whether people can actually respond | Within the policy, consistently |
| Distribution across people | Swaps and specialisation can concentrate the load | Roughly even |
| Time spent on follow-up | Incidents need postmortems and fixes | Enough; a rushed follow-up repeats the incident |
The limit of about two incidents per shift is not arbitrary. Handling an incident properly includes mitigation, investigation, writing it up and starting the fix, which takes hours. Beyond that rate, engineers can only fight fires, the causes are never addressed, and the load rises further. When a rotation is consistently over the limit, treat it as an emergency for the team: pause feature work, fix the top sources of pages, and consider returning the pager for the worst service to its developers, as the first module described.
The opposite problem is real too. A rotation that is almost never paged loses its skills: people forget the tools, and the runbooks rot. Keep the muscle alive with regular drills and game days, covered in the chaos engineering module.
- Protect sleep. After a night with a long incident, the engineer starts late or takes the day. A tired responder is a risk to the service.
- Reduce the on-call person's project load for the week. They are interrupt-driven by design, so give them small, interruptible work such as runbook fixes and alert tuning.
- Make it psychologically safe. People who fear blame hesitate, hide mistakes and avoid escalating. A blameless culture is a precondition for fast response.
- Provide the tools: a laptop, mobile data, access that works, and a quiet way to get on a call. Test them before the first shift, not during the first incident.
Preparing new responders
Nobody should carry a pager alone before they are ready, and being ready is something you can build deliberately instead of hoping for.
- Learn the system. Architecture walkthrough, the service's SLOs and dashboards, where the runbooks live, and how to deploy and roll back. Have the newcomer deploy and roll back for real in staging.
- Read history. Work through the last several postmortems. They show how the system actually fails, which the architecture diagram never does.
- Shadow. The newcomer receives the same pages as the primary and follows along, without responsibility. They see real incidents handled.
- Reverse shadow. The newcomer is primary, and an experienced engineer shadows them, ready to step in. Confidence comes from handling a real page with a safety net.
- Practise. Run a drill such as "wheel of misfortune", in which a past incident is replayed as a role-play and the trainee talks through what they would check and do.
- Go solo, with a named secondary who expects to be called, and review the first shift together.
An on-call readiness checklist makes the expectations explicit: access to every system verified, paging app installed and tested with a real test page, runbooks read, a rollback performed, escalation contacts known. It is also an excellent test of your documentation. Every question a new engineer has to ask is a gap in a runbook.