Security incident response

The lifecycle and why preparation is most of it

The standard model (from NIST's incident handling guide) has six phases: preparation, detection and analysis, containment, eradication, recovery, and post-incident activity. Teams that do well in the middle four did the work in the first one: logs are centralised and retained, roles are defined, playbooks exist for the common scenarios, contact details for legal, communications, the cloud provider and law enforcement are written down, and everyone has practised at least once. Improvising all of that at 3 a.m. with an attacker inside is the failure mode.

• Playbooks for the likely incidents: leaked credential, phished account, malware on an endpoint, web application compromise, ransomware, insider misuse, cloud account takeover. One page each: how to confirm, how to contain, who to call.
• Roles: incident commander (decisions and coordination), investigation lead (technical analysis), communications lead (internal and external updates), scribe (timeline). Same people can hold several roles in a small team, but the roles must be named.
• Access ready: break-glass credentials for investigators, the ability to isolate hosts and revoke sessions quickly, forensic tooling installed before it is needed.
• Contacts: legal counsel, cyber insurance, the cloud provider's abuse or security contact, a forensics firm on retainer, relevant regulators' notification routes.
• Practice: a tabletop exercise per quarter, walking through a scenario and finding the gaps in the playbooks.

The first hour: declare, preserve, scope

When an alert or a report looks real, declare the incident early and loudly; it is cheap to stand down a false alarm and expensive to lose an hour. Assign the roles. Start the timeline: a shared document where the scribe records every observation, decision and action with a UTC timestamp. Then, before touching anything, preserve evidence: snapshot the affected instances and volumes, export the relevant logs to a separate location, capture running process and connection lists, and note the hashes of anything you collect.

EVID=/mnt/evidence/$(hostname)-$(date -u +%Y%m%dT%H%M%SZ)
sudo mkdir -p "$EVID" && cd "$EVID" || exit 1

sudo ps -eo pid,ppid,user,lstart,etime,cmd --sort=lstart > processes.txt
sudo ss -tunapee > sockets.txt
sudo lsof -nP > open_files.txt 2>/dev/null
last -F > logins.txt; sudo lastb -F > failed_logins.txt 2>/dev/null
sudo cat /etc/passwd /etc/group > accounts.txt
sudo crontab -l -u root > cron_root.txt 2>/dev/null; ls -la /etc/cron* /var/spool/cron > cron_files.txt 2>&1
sudo systemctl list-units --type=service --all > services.txt
sudo find / -xdev -mmin -1440 -type f ! -path '/proc/*' -printf '%TY-%Tm-%Td %TH:%TM %u %p\n' > modified_24h.txt 2>/dev/null

sha256sum ./* > SHA256SUMS
# and in the cloud console or CLI: snapshot the instance's volumes now, tag them "evidence", restrict access

Then scope before you act. The alert that fired is one footprint; the attacker has usually been in longer and wider. Pivot from what you know: which credential, which host, which IP, which time window. Search the central logs for every use of that credential, every connection from that IP, every host that talked to the compromised one, every new user, key, role or scheduled task created in the window. Write each new finding in the timeline. Containment that misses part of the footprint tells the attacker you are coming and leaves them a way back in.

Containment: stop the bleeding, keep the evidence

Containment stops the attacker from doing more damage while you finish understanding the incident. Choose actions that are reversible where possible and quiet where it matters: revoking a credential and rotating it, isolating a host at the network layer (security group with no rules, or a quarantine VLAN) while keeping it running for analysis, blocking an IP or an ASN at the edge, disabling a user's sessions and tokens, taking a compromised service off the load balancer. For an active, capable intruder, coordinate so that all footholds are cut at once; cutting one at a time is a signal.

Record every containment action in the timeline with who did it and when. Some actions have side effects for customers (a blocked range, an offline service); the incident commander weighs those with the business, but the default when data is leaving is to stop it now and apologise for the outage later.

Eradication and recovery

Eradication removes the attacker's presence: the malware, the webshell, the added user, the persistence in cron or a systemd unit, the forwarding rule in the mailbox, the extra IAM role, the SSH key in authorized_keys. Because you rarely find everything by hand, prefer rebuilding from known-good sources over cleaning: redeploy the service from the pipeline, re-create the host from the image, restore data from a backup taken before the intrusion (verify the backup is clean and the restore actually works). Close the entry point: patch the vulnerability, fix the misconfiguration, enforce the MFA that was missing.

Recovery brings services back with heightened monitoring: watch for the indicators you collected (IPs, domains, hashes, user agents, the attacker's habits) and for any repeat of the entry technique. Rotate every credential the attacker could have reached, not just the one they used. Confirm with the detection team that the alerts that should fire on a return actually fire. Only when the timeline shows a clear entry, full scope, complete containment and eradication, and a clean recovery does the incident close.

sudo find / -xdev -name authorized_keys -exec sh -c 'echo "== $1"; cat "$1"' _ {} \; 2>/dev/null
sudo ls -la /etc/cron.d /etc/cron.daily /var/spool/cron/crontabs 2>/dev/null
sudo systemctl list-unit-files --state=enabled | grep -v '^unit'
sudo find /etc/systemd/system /lib/systemd/system -newer /etc/hostname -name '*.service' 2>/dev/null
sudo grep -rl 'curl\|wget\|base64 -d' /etc/rc.local /etc/profile.d /home/*/.bashrc /root/.bashrc 2>/dev/null
sudo find / -xdev -perm -4000 -newer /etc/hostname -type f 2>/dev/null

Communication, notification and the postmortem

Inside the company, the communications lead sends short, regular updates (every hour during the active phase) on a fixed channel: what is known, what is being done, what is needed. Speculation stays out of writing; facts and next steps go in. Outside the company, notification is often a legal obligation with a clock: personal data breaches under GDPR must be reported to the regulator within 72 hours of awareness, and many jurisdictions and contracts have their own rules. Involve legal counsel early, keep the timeline accurate because it becomes the evidence of what you knew when, and tell affected customers what happened, what it means for them and what you have done — plainly, without minimising.

After recovery, run a blameless postmortem exactly as the SRE track describes: the timeline, the root cause and the contributing causes, what went well, what did not, and actions with owners and dates. Security incidents add specific questions: how long was the attacker present before detection (dwell time), which detection should have fired earlier, which control would have stopped the entry, and what evidence was missing. Then update the playbooks and detections, and schedule the tabletop that rehearses the next one.