You have added timeouts, retries, circuit breakers, redundancy across zones, burn-rate alerts and runbooks. Do they work? The honest answer, for most teams, is that nobody knows, because those mechanisms only run when something breaks, and the first real test is a real outage at a time you did not choose. Chaos engineering is the practice of choosing the time yourself. You inject a failure deliberately, in a controlled way, to find out whether the system behaves as you believe it does.
- Explain chaos engineering as controlled experimentation, and distinguish it from breaking things at random
- Design an experiment with a steady state, a hypothesis, a bounded blast radius and abort conditions
- Inject common faults using standard Linux and Kubernetes tooling: latency, packet loss, process and node failure, resource pressure
- Plan and run a game day that tests people and process as well as the system
- Decide when a team is ready for chaos engineering, and progress safely from staging to production
Experiments, not vandalism
The name is unfortunate. Chaos engineering is not about causing chaos. It is the disciplined use of experiments to build confidence that a system can withstand turbulent conditions in production. Netflix popularised it when it moved to the cloud and built Chaos Monkey, a tool that terminated production instances at random during working hours. The purpose was not destruction. It was to make instance failure so routine that every team had to build services that survived it.
The difference from testing is one of intent. A test checks a known property and returns pass or fail: given this input, is that the output? An experiment explores something you are unsure of, and produces new knowledge: if the cache disappears, what actually happens? Complex distributed systems have behaviours that nobody designed and nobody can predict from reading code, such as retry storms, failovers that take four minutes instead of four seconds, and a fallback that calls the very dependency it is supposed to replace. Experiments are how you find them before your users do.
Every experiment has one of two outcomes, and both are good. Either the system behaves as predicted, and you have earned justified confidence. Or it does not, and you have found a weakness at a time of your choosing, with everyone awake, a rollback ready and the blast radius limited. The only bad outcome is the one you were heading for anyway: discovering the weakness in a real incident.
If you already know that an experiment will cause an outage, do not run it. You have nothing to learn, only damage to do. Fix the known weakness first, and then run the experiment to confirm the fix.
The method
The Principles of Chaos Engineering describe a method close to any scientific experiment.
- Define the steady state. Choose a measurable output that shows the system is healthy from the user's point of view. Your SLIs are exactly this: checkout success ratio, p99 latency, orders per minute. Not CPU.
- Form a hypothesis that the steady state will continue under a specific failure. "If one of three cache nodes is terminated, checkout success ratio stays above 99.9% and p99 stays under 500 ms."
- Introduce a real-world event: a server dies, a dependency slows down, a zone becomes unreachable, a disk fills, a certificate expires, a clock drifts.
- Limit the blast radius. Begin with the smallest scope that can teach you something: one instance, one percent of traffic, one test customer.
- Define abort conditions and a stop button before starting. If the steady-state metric falls below a stated level, the experiment stops at once and the fault is removed.
- Run it, observe, and compare what happened with the hypothesis.
- Fix what you found, then run it again. Once it passes reliably, widen the scope, and eventually automate it so that it runs continuously.
# Chaos experiment: loss of one cache node
Owner: Priya S. Date: 2026-09-24 10:00 UTC Environment: production
Observers: checkout on-call, platform on-call Channel: #chaos-2026-09-24
## Steady state
- Checkout success ratio >= 99.9% (5 min window)
- Checkout p99 latency < 500 ms
## Hypothesis
Terminating 1 of 3 cache nodes causes a brief rise in latency (p99 < 800 ms
for under 2 minutes) and no rise in errors. Clients reconnect automatically.
## Method
Delete pod cache-1. Observe for 15 minutes.
## Blast radius
One cache node. All checkout traffic may see higher latency. No data at risk.
## Abort conditions (any one)
- Success ratio < 99.5% for 2 minutes
- p99 > 2 s for 2 minutes
- Any unrelated incident is declared
## Rollback
The StatefulSet recreates the pod automatically. If it has not recovered in
5 minutes: scale to 4 replicas. Runbook: cache-node-down.md
## Result
(observed behaviour, deviations from hypothesis, action items with owners)Writing the plan is half the value. Teams regularly discover while filling in the hypothesis that nobody knows what is supposed to happen, or while filling in the rollback section that there is no way to undo the fault. Both are findings, and they cost nothing.
What to break, and how
Choose experiments from evidence, not imagination. The best sources are your own postmortems (would we survive that again?), the dependency map from the embracing-risk module (what happens when each hard dependency fails?), and the defences from the failure-modes module that have never been exercised.
| Fault | What it tests | Typical finding |
|---|---|---|
| Kill a process, pod or instance | Redundancy, restart behaviour, load redistribution | Slow start-up; dropped in-flight requests; survivors overloaded |
| Add network latency | Timeouts, deadlines, user experience when slow | No timeout configured; threads exhausted; slow is worse than down |
| Drop packets or block a dependency | Retries, circuit breakers, fallbacks | Retry storm; the fallback was never implemented |
| Exhaust CPU, memory or disk | Limits, autoscaling, alerting, eviction | Alerts do not fire; noisy neighbour takes down the node |
| Drain a node, or fail a zone | Scheduling, capacity headroom, zone independence | Not enough capacity; everything depended on one zone |
| Expire a certificate, revoke a credential | Rotation, monitoring of expiry | Nobody knew where the certificate was used |
| Return errors or bad data from a dependency | Validation and error handling | One malformed response crashes the consumer |
kubectl delete pod cache-1 -n shop # abrupt pod loss
kubectl delete pod -n shop -l app=checkout --field-selector spec.nodeName=node-a
kubectl cordon node-a # no new pods here
kubectl drain node-a --ignore-daemonsets --delete-emptydir-data # evict everything
kubectl uncordon node-a # undo
kubectl scale deployment/recommendations --replicas=0 -n shop # dependency gonetc qdisc add dev eth0 root netem delay 200ms 50ms # 200 ms latency, 50 ms jitter
tc qdisc change dev eth0 root netem loss 5% # switch to 5% packet loss
tc qdisc show dev eth0 # what is active?
tc qdisc del dev eth0 root # REMOVE the fault
iptables -A OUTPUT -p tcp --dport 5432 -j DROP # black-hole the database
iptables -D OUTPUT -p tcp --dport 5432 -j DROP # undo
stress-ng --cpu 4 --timeout 120s # burn 4 cores for 2 minutes
stress-ng --vm 2 --vm-bytes 1G --timeout 120s # memory pressure
fallocate -l 10G /var/tmp/fill.img # fill a disk (rm the file to undo)Know how to undo every fault before you inject it, and write the undo command into the plan. tc and iptables rules persist until they are removed or the machine reboots, and a forgotten packet-loss rule is a baffling incident three days later. Prefer faults with a built-in time limit, such as --timeout, wherever you can.
Purpose-built tools make this safer and repeatable. Chaos Mesh and LitmusChaos are open-source projects that define experiments as Kubernetes resources, with scheduling, scoping by label and automatic clean-up. Service meshes can inject delays and errors into a chosen share of requests, which gives very precise control over blast radius, and the large cloud providers offer managed fault-injection services. Learn the mechanics with the basic commands first, so that you understand what the tools are doing on your behalf.
Game days
A game day is a scheduled exercise in which a team practises responding to failure. It tests more than the software: it tests whether alerts fire and reach the right person, whether the dashboards show the problem, whether the runbook is correct, whether people can get access, and whether the incident process from the earlier module holds up. Those are the parts that most often fail, and no automated experiment exercises them.
- Plan. Choose one or two scenarios and write experiment plans for them. Pick a date in working hours, not before a weekend, holiday or launch. Tell everyone who could be affected, including support and the owners of dependencies.
- Assign roles. A coordinator runs the day and holds the stop button. An operator injects the faults. Responders handle the incident as if it were real. Observers take notes on what happens, including timings.
- Decide what the responders know. In an announced exercise they know the scenario. In a surprise exercise they know only that something will happen. Surprise teaches more about detection, and should come only after the team is comfortable.
- Run it. Inject the fault, and let the normal process work: the alert, the page, the declaration, the roles, the runbook. The coordinator watches the abort conditions throughout.
- Debrief straight away, while it is fresh, in the blameless style of a postmortem. What did we expect, and what happened? How long did detection take? What was confusing? What would have gone badly at 03:00?
- Turn findings into tickets with owners, and schedule a re-run of anything that failed.
Game days are also the best training there is. They are where new on-call engineers gain experience safely, where the reverse shadowing from the on-call module happens, and where runbooks get tested by someone other than their author. A lighter format that needs no infrastructure is the tabletop exercise: a facilitator describes a scenario and reveals information as the team asks for it, and the team talks through what it would do.
Do not limit the scenarios to software. Practise the loss of a key person: can the team restore from backup when the one engineer who has done it before is on holiday? Practise the loss of a tool: can you respond when the chat system or the monitoring is itself what has failed?
Readiness, and the road to production
Chaos engineering is not where reliability work begins. It verifies defences, so there must be some. A team that is already losing the battle against ordinary incidents has nothing to gain by adding artificial ones. Check the prerequisites first.
- Observability. You can see the steady state in near real time. Without it you cannot tell what the experiment did, or when to stop.
- Known weaknesses fixed. Do not spend an experiment proving what you already know.
- A way to stop and undo every fault, quickly.
- Basic resilience in place: more than one instance, health probes, timeouts. Otherwise the result is certain.
- An incident process that works, because an experiment can become a real incident.
- Agreement from the people affected: management, neighbouring teams, support. Surprising your colleagues with an outage, even a deliberate one, destroys the trust the practice depends on.
- Error budget to spend. An experiment that goes wrong consumes budget, which is a legitimate use of it. If the budget is exhausted, wait.
Then progress gradually, widening the scope only as each stage passes.
| Stage | Where | Why |
|---|---|---|
| 1 | Tabletop | Finds gaps in understanding and process at no risk |
| 2 | Development or staging | Learn the tooling; find the obvious failures |
| 3 | Production, one instance or a sliver of traffic, in working hours | Staging never matches production's scale, data, configuration and traffic |
| 4 | Production, wider scope: a node, a zone, a dependency | Tests capacity headroom and zone independence for real |
| 5 | Automated and continuous | Guards against regression, because systems change every day |
The move to production makes people nervous, with reason. But production is the only environment with production's traffic patterns, data volumes, configuration and dependencies, so it is the only place where some weaknesses can be seen. A controlled experiment on one percent of traffic, at ten in the morning, with the team watching and a stop button in hand, is far safer than the uncontrolled version that the same weakness will otherwise deliver.
The final stage matters because resilience decays. A fallback that worked in March is broken by an unrelated refactor in June, and nobody notices, because it never runs. Experiments that run continuously, in the pipeline or on a schedule, turn resilience from a belief into a property that is checked, in the same way that unit tests do for correctness.
Start smaller than feels useful. The first experiment can be deleting one pod of a stateless service in staging and watching the dashboard. It takes ten minutes, it is almost certain to reveal something, such as a few dropped requests during shutdown, and it builds the habit and the trust that make the ambitious experiments possible later.