The first minutes of a major incident are usually chaos: eight engineers in a call all debugging different theories, three people making changes nobody else knows about, executives asking for updates in four channels, and nobody sure who is in charge. Technical skill does not fix that. Structure does. This module teaches the incident management practices, borrowed from emergency services, that let a group of people work on a crisis without getting in each other's way.
- Decide when to declare an incident, and assign a severity using a written matrix
- Describe the incident commander, operations, communications and scribe roles, and hand them over cleanly
- Prioritise mitigation over diagnosis, and choose the quickest safe way to stop the impact
- Communicate clearly under pressure, with regular updates for responders, stakeholders and customers
- Close an incident properly, preserving what the postmortem will need
Declare early, and say how bad it is
An incident is an unplanned event that degrades a service, or threatens to, and needs a coordinated response. The most common mistake is declaring too late: an engineer spends forty minutes investigating alone, hoping it turns out to be nothing, while the impact grows. Declaring an incident costs almost nothing, and standing one down is easy. Give people an explicit rule so they do not have to agonise over it.
- Users are affected, or will be soon, and there is no fix already in progress that will land within minutes.
- You need help from a second person or another team.
- You have been investigating for fifteen minutes and do not yet understand the problem.
- Customers, support or management are already asking about it.
- When in doubt, declare. A false alarm is a free drill.
Severity determines how much of the organisation is mobilised. Write the definitions down in advance, in terms of user and business impact, never in terms of technical cause, so that at 03:00 the decision is a lookup and not a debate.
| Severity | Definition | Response | Updates |
|---|---|---|---|
| SEV1 | Critical user journey down or data at risk, for many or all users | All hands; IC assigned at once; executives informed; status page | Every 30 minutes |
| SEV2 | Major feature degraded, or a significant share of users affected | On-call plus the teams needed; IC assigned; status page if customer-visible | Every hour |
| SEV3 | Minor impact, a workaround exists, or an internal service is affected | On-call handles it in working hours | As things change |
| SEV4 | No user impact yet; something needs attention | A ticket | None |
Severity can change in both directions as you learn more. Err on the high side at the start, since it is easier to stand people down than to summon them late. Declaring should be one action: a chat command or a button that opens a dedicated channel, starts a call, pages the right people and begins a timeline.
Roles: who does what
The structure comes from the Incident Command System, developed by emergency services in the United States for coordinating responses to wildfires and used since for emergencies of every kind. Its central idea is a clear separation of responsibilities with a single person in charge, so that everyone knows what they own and whom they report to.
| Role | Responsible for | Explicitly not |
|---|---|---|
| Incident commander (IC) | The overall response: holds the big picture, assigns roles, sets priorities, makes decisions, decides when it is over | Debugging. An IC with their hands on a keyboard has stopped commanding. |
| Operations / tech lead | Investigating and changing the system, and directing other responders who do | Talking to stakeholders |
| Communications lead | Updates to stakeholders, support, customers and the status page, on a regular rhythm | Being in the technical weeds |
| Scribe | A timestamped record of what was observed, decided and done | Interpreting or judging it |
In a small incident one person may hold several roles, and the first responder holds all of them until others arrive. What matters is that each role is explicitly held by someone. As soon as more than two or three people are involved, split them, starting with separating the IC from operations.
- The IC does not need to be the most senior person, or the greatest expert on the system. They need to stay calm, keep the overview, and be good at asking questions and delegating. It is a trained skill, and many organisations keep a dedicated IC rotation.
- Only operations changes production, and every change is announced first: "I am about to restart the payment workers". Unannounced changes by well-meaning helpers are how one incident becomes two.
- Senior leaders who join the call are observers unless the IC gives them a role. Their questions go to the communications lead, not to the engineers who are debugging.
- Anyone may be asked to leave the call, and people who are not needed should offer to. A large audience slows everything down.
Hand over roles explicitly and out loud. "Marco, you are now incident commander. Summary: checkout errors at 30%, caused by the 14:02 deploy, rollback is in progress, Lena has comms. Do you accept?" "I accept. I am now IC." Then announce it in the channel. Long incidents need handovers, because nobody commands well after four hours without a break.
Mitigate first, diagnose later
The engineer's instinct is to understand the problem. The responder's job is to stop the bleeding. Every minute spent finding the root cause while users are failing is a minute of error budget spent on curiosity. Ask first what the fastest safe action is that reduces the impact, even if you do not yet know why it works.
| Mitigation | Use when | Note |
|---|---|---|
| Roll back the last change | The trouble began just after a deploy or config change | The default first move. Most incidents follow a change. |
| Switch off a feature flag | A specific new feature is implicated | Faster and narrower than a rollback |
| Fail over or drain | One zone, region, node or replica is unhealthy | Move traffic away from the bad part |
| Scale up or out | The system is saturated by load | Buys time; does not fix a leak |
| Shed load or degrade | Demand exceeds capacity and you cannot add more | Rate-limit, serve cached content, disable expensive features |
| Restart | A process is stuck or has leaked resources | Capture diagnostics first if you can; it destroys evidence |
| Block bad traffic | One client or an attack is causing the problem | At the edge, not in the application |
The first diagnostic question is always "what changed?" Check the deploy annotations on the dashboard, the deployment history, config and flag changes, infrastructure changes, certificate expiries, and what your dependencies and cloud provider are reporting. If a change lines up with the start of the symptoms, roll it back without waiting to understand it. You can work out why tomorrow, from the evidence, with the service healthy.
kubectl rollout history deployment/checkout -n shop # what was deployed, and when?
kubectl get pods -n shop -l app=checkout -o wide # restarts? pending? one node?
kubectl get events -n shop --sort-by=.lastTimestamp | tail -20
kubectl logs deploy/checkout -n shop --since=15m | grep -iE 'error|timeout|refused' | tail -30
kubectl top pods -n shop -l app=checkout
# the quickest safe mitigation after a bad deploy
kubectl rollout undo deployment/checkout -n shop
kubectl rollout status deployment/checkout -n shop- One change at a time, announced beforehand, recorded afterwards, and followed by a look at the graphs. Several simultaneous changes make it impossible to know what helped.
- State hypotheses out loud and test them. "I think the database is saturated; if so, connections will be at the limit. Checking." It keeps the group aligned and prevents anchoring on the first theory.
- Timebox each line of investigation. If fifteen minutes on one theory yields nothing, step back and try another approach, or escalate.
- Preserve evidence where it is cheap: a heap dump, a copy of the logs, a snapshot of the broken pod before it is deleted. It is what the postmortem will need.
Beware of fixes that make things worse. Restarting everything at once produces a thundering herd against a cold cache and a struggling database. Bring things back gradually, and watch the dependencies as well as the service.
Communicating under pressure
Silence is interpreted as incompetence. Stakeholders who do not hear anything will interrupt the responders to ask, and customers who see nothing on the status page assume you have not noticed. Regular, predictable updates protect the people doing the work, even when the update is that nothing has changed.
Use one channel for the incident and one call for the responders, and keep decisions in writing in the channel so that people joining late can catch up without asking. The communications lead posts updates at the interval set by the severity, and each update states when the next one is due.
INCIDENT UPDATE #3 SEV2 INC-2304 14:45 UTC
Status: Mitigating
Impact: About 30% of checkout attempts fail with an error. Browsing and
login are unaffected. Started 14:04 UTC.
What we know: Errors began two minutes after deploy 7f3a9c1 to checkout.
Payment provider reports no problems.
What we are doing: Rolling back to the previous version (started 14:41).
Next update: 15:15 UTC, or sooner if anything changes.
IC: Marco T. Ops: Priya S. Comms: Lena K. Channel: #inc-2304Investigating - 14:20 UTC
Some customers are seeing errors when completing a purchase. We are
investigating and will post an update by 14:50 UTC.
Identified - 14:45 UTC
We have identified the cause and are rolling out a fix. Some purchases may
still fail. Items in your basket are not affected. Next update by 15:15 UTC.
Resolved - 15:10 UTC
Purchases are working normally again. Between 14:04 and 15:02 UTC, around a
third of purchase attempts failed. No orders or payments were lost. We are
sorry for the disruption and will publish a review of what happened.- Describe impact in the user's terms: what they cannot do, since when, and how many are affected. Not "pods are crash-looping".
- Separate what you know from what you suspect. Never guess in writing, and never promise a time for the fix. Promise a time for the next update and keep it.
- For customers, be honest, specific and free of jargon. Say what is affected, what is not, and whether their data is safe, which is the question they care about most.
- On the call, use closed-loop communication. The IC asks a named person to do something, that person repeats it back, and reports when it is done. "Priya, please roll back checkout." "Rolling back checkout now." "Rollback complete, error rate is falling."
- Address people by name. "Can someone check the database?" means nobody checks the database.
Closing the incident
An incident moves through recognisable states: investigating, identified, mitigating, monitoring, resolved. Do not jump from mitigating straight to resolved. After the fix, watch the SLIs for a period that suits the system, often fifteen to thirty minutes, to be sure that it holds and that nothing has backed up behind it: queued jobs, retries, cold caches.
The IC declares the incident resolved, and then the closing tasks matter as much as the response.
- Announce the resolution in the channel and post the final status page update.
- List every temporary change made during the response: raised limits, disabled flags, silenced alerts, manual overrides. Assign an owner to reverse or formalise each one.
- Save the evidence: the channel history, the scribe's timeline, dashboard screenshots or links with fixed time ranges, relevant logs and traces before they age out.
- Assign a postmortem owner, and set a date, within a few days, while memories are fresh.
- Thank the responders, and make sure anyone who was up during the night gets rest. Then stand the call down.
Incident response is a skill, and skills fade without practice. Most teams have few real SEV1 incidents, which is good news that produces rusty responders. Run drills: replay a past incident as a tabletop exercise, practise the IC role on small incidents where the stakes are low, and rehearse the declare-and-assemble mechanics so that the tooling is familiar. The goal is that on the day it matters, the process is the one thing nobody has to think about.