An incident is an expensive lesson, and the fee is paid whether or not you learn anything. A postmortem is how you collect what you paid for. Done well, it turns one bad afternoon into permanent improvements to systems and process. Done badly, it produces a document that names a culprit, recommends being more careful, lists ten action items that are never completed, and teaches everyone to hide their mistakes next time. This module covers how to run the good kind.
- Explain what blameless means, and why blame makes systems less safe
- Decide which incidents need a postmortem, and who should own it
- Reconstruct an accurate timeline and analyse contributing factors instead of a single root cause
- Write action items that are specific, owned, prioritised and actually completed
- Run a review meeting and share the results so that the whole organisation learns
Why blameless
A postmortem, also called an incident review or retrospective, is a written record of an incident: what happened, the impact, what was done about it, why it happened, and what will change. Blameless means it is written on the assumption that everyone involved acted with good intentions and made reasonable decisions given the information, tools and time they had.
This is not kindness for its own sake. It is the only approach that works. If the engineer who ran the wrong command is punished, three things follow. The next person who makes a mistake hides it, and you learn about it later and at greater cost. People become cautious to the point of paralysis. And the real finding is lost, because "Alex was careless" ends the inquiry at the exact moment it should begin. Why was that command possible? Why did staging look identical to production in the terminal? Why was there no confirmation, no dry run, no review?
People will always make mistakes. A system that depends on nobody ever making one is a badly designed system, and that design is what a postmortem examines. The person closest to the failure is your best source of information about how the system really behaves, and they will only tell you if it is safe to.
| Blaming language | Blameless language |
|---|---|
| Alex ran the migration against production by mistake | The migration was run against production. The tool takes the target from an environment variable that was still set from an earlier session, and gives no indication of which database it is connected to. |
| The on-call engineer failed to notice the alert | The alert was delivered to a channel that receives about 200 messages a day. It was seen 40 minutes later. |
| QA should have caught this | The test suite does not cover requests with more than 50 items. Such requests are about 0.2% of production traffic. |
Blameless does not mean unaccountable. People remain accountable for taking part honestly, for completing the actions they own, and for their professional conduct. Blamelessness covers honest mistakes made while doing the job, which is nearly all of them. Naming individuals is usually unnecessary; roles such as "the on-call engineer" work better.
When to write one, and who does it
Define the triggers in advance, so that whether to hold a review is never a political question. Typical criteria follow.
- Any SEV1 or SEV2 incident.
- User-visible downtime or degradation beyond a set threshold.
- Data loss of any kind, however small.
- An incident that consumed more than a set share of the error budget, such as 20%.
- Any incident that needed a manual intervention by on-call, such as a rollback or a traffic reroute.
- A monitoring failure: the problem was found by a customer or by luck, not by an alert.
- A near miss, where luck prevented an outage. These are the cheapest lessons you will ever be offered.
- Anyone involved asks for one.
Appoint one owner, usually someone who took part in the response but not necessarily the person at the centre of events. The owner gathers input from everyone involved, drafts the document, and drives the review. Aim to circulate a draft within about five working days. Sooner is better, because memories fade and blur quickly, and chat logs and metrics age out of retention.
Scale the effort to the incident. A SEV3 may need a half-page note and one action. A SEV1 with customer impact deserves a thorough document, a review meeting, and often a version written for customers. A lightweight process that is always followed is better than a heavy one that people avoid.
The document
# Postmortem: Checkout failures after deploy (INC-2304)
Date: 2026-09-17 Severity: SEV2 Status: Draft | In review | Final
Owner: Priya S. Reviewers: checkout team, platform team
## Summary
Two or three sentences a busy person can read: what broke, for whom, for how
long, and what fixed it.
## Impact
- Users: about 31% of checkout attempts failed between 14:04 and 15:02 UTC.
- Business: roughly 1,900 failed purchases; 140 support contacts.
- SLO: consumed 38% of the 28-day error budget.
- Data: no orders or payments were lost.
## Timeline (UTC)
14:02 Deploy 7f3a9c1 reaches 100% of checkout pods.
14:04 Error ratio rises from 0.05% to 31%.
14:09 CheckoutErrorBudgetFastBurn pages the on-call engineer. <- detection
14:14 Incident declared, SEV2. IC assigned.
14:41 Rollback started, after ruling out the payment provider.
15:02 Error ratio back to normal. <- mitigation
15:20 Incident resolved after monitoring.
## Detection
How did we find out? How long did it take? Could it have been faster?
## Response
What went well, what was difficult, where we were lucky.
## Contributing factors
The conditions that, together, made this incident possible and made it last.
## Action items
| Action | Type | Priority | Owner | Due | Ticket |
## Lessons learned
What went well. What went badly. Where we got lucky.
## Supporting material
Links to dashboards (fixed time ranges), the incident channel, traces, the diff.Three parts deserve particular care. The impact should be stated in numbers and in users' terms, because it determines how much investment the follow-up justifies, and because vague impact statements make every incident sound the same.
The timeline is the factual spine. Build it from sources that carry timestamps, such as chat history, alert and deploy logs, dashboards and the scribe's notes, and not from memory, which reliably reorders and compresses events. Record observations, decisions and actions, and mark the key moments: when the fault was introduced, when impact began, detection, declaration, mitigation and resolution. The intervals between them are the numbers worth tracking: time to detect, time to mitigate and time to resolve.
Where we got lucky is the section teams leave out and should not. "An engineer happened to be looking at that dashboard", or "it occurred at a quiet time of day", marks a place where the outcome could have been much worse and nothing in your system prevented it.
From root cause to contributing factors
The phrase root cause suggests that an incident has a single origin which, once removed, makes the system safe. Real incidents in complex systems rarely work like that. They need several things to be true at once: a latent bug, a gap in testing, a missing alert, a confusing tool, a deploy that went to all instances at the same time, a slow rollback. Remove any one and the incident does not happen, or is far smaller. Each of them is a contributing factor, and each is an opportunity.
The five whys technique, asking "why?" repeatedly, is a useful prompt for getting past the first answer. Its weakness is that it follows a single chain and tends to end at a person or at the last step. Ask it as a tree instead, with several branches.
Why did 31% of checkouts fail?
-> The new version rejects baskets with more than 50 items.
Why was that not caught before release?
-> No test covers large baskets; staging data has none.
Why did it reach every user at once?
-> The deploy goes to 100% in one step; there is no canary stage.
Why did it last 58 minutes?
-> Detection took 5 minutes (good), but mitigation took another 53.
Why so long?
-> 27 minutes after declaration went on investigating the payment provider.
Why?
-> The dashboard has no deploy annotations, so the link to the
14:02 deploy was not visible.
-> The rollback itself needed a manual approval from someone who was
in a meeting.Useful questions move the inquiry away from individuals and towards the system.
- What made this action seem reasonable to the person at the time?
- What information did they have, and what were they missing?
- What did the tools make easy, and what did they make hard?
- Which defences were supposed to catch this, and why did each one fail?
- How did we detect it, and what would have detected it earlier?
- What slowed the response down, and what made it possible to recover at all?
- Has this happened before? If so, what became of the earlier action items?
Watch for hindsight bias. Once you know the outcome, the warning signs look obvious and the right decision looks easy. They were not, to someone in the middle of events with incomplete information and twenty other signals competing for attention. If the postmortem says someone "should have known", rewrite it to describe what they could actually see.
Action items that get done
The action items are the return on the whole exercise, and also where most postmortems fail. A review that ends with fifteen vague intentions and no owners has achieved nothing. Every action item should pass a simple test.
| Quality | Weak | Strong |
|---|---|---|
| Specific and verifiable | Improve monitoring | Add deploy annotations to the checkout dashboard from the CD pipeline |
| Owned | The team will look into it | Owner: Marco T. |
| Dated and tracked | Soon | Due 2026-10-01, ticket SHOP-4412 |
| Systemic | Be more careful with large baskets | Add a test for 100-item baskets and seed staging with large baskets |
| Prioritised | A list of twelve equal items | Two P1 items that would have prevented or halved this incident |
Classify each action by what it does, and check that you have covered more than one category. Teams drift towards prevention alone, although faster detection and mitigation pay off against every future incident, including the ones you cannot predict.
- Prevent: stop this class of fault from occurring. Tests, validation, safer defaults, removing the hazard.
- Detect: find it sooner. Alerts, dashboards, synthetic checks.
- Mitigate: reduce impact and duration. Canary releases, faster rollback, feature flags, graceful degradation.
- Process: runbooks, training, handoffs, escalation paths.
Be suspicious of three kinds of action. "Be more careful" and "remind everyone" rely on human vigilance, which is precisely what failed. "Add a manual approval step" adds friction and rarely adds safety. And a long list dilutes attention: prefer the two or three items with the most leverage, and ticket the rest at a lower priority, honestly labelled.
Then follow through. Track postmortem actions in the same system as other work, review open ones at a regular meeting, and report the completion rate. When the error budget policy says that reliability work takes priority, these are the items it means. A repeat incident whose earlier action items were never done is the clearest sign of a broken process, and it deserves a postmortem of its own.