Security logging, SIEM & detection

What to log, and what a log line needs

Security logging answers four questions after the fact: who, did what, to which resource, from where, and when. Logs that lack any of those are much less useful. Prioritise the sources where attacker behaviour shows up first.

Emit application security events deliberately, in structured form, with a stable event name: auth.login.failed, authz.denied, data.export.requested, admin.user.role_changed. Include the acting identity, the target, the outcome and the request id that ties it to the rest of the trace. Never include secrets, session tokens or full card numbers; redact at the source.

import json
import logging
from datetime import datetime, timezone

sec = logging.getLogger("security")


def security_event(name: str, actor: str, target: str, outcome: str, request_id: str, **extra):
    sec.info(json.dumps({
        "ts": datetime.now(timezone.utc).isoformat(),
        "event": name,            # e.g. "authz.denied", "data.export.requested"
        "actor": actor,           # user id or service identity, never a token
        "target": target,         # resource identifier
        "outcome": outcome,       # "success" | "denied" | "error"
        "request_id": request_id,
        **extra,
    }))


security_event("authz.denied", actor="user:1042", target="order:99817", outcome="denied",
               request_id="7f3c...", src_ip="203.0.113.9")

Centralise, protect, retain

Logs on the host that produced them are evidence in the attacker's custody. Ship them immediately (an agent such as Fluent Bit, Vector or the cloud provider's agent) to central storage that the source systems can write to but not delete from: object storage with write-once policies, or a log platform with immutability and a separate administrative boundary. A SIEM (security information and event management system) is a log platform specialised for this: normalised schemas across sources, correlation rules, case management and long retention. Open-source and cloud-native options exist; the key property is that the security team's data cannot be altered by the people or systems it monitors.

pipeline:
  inputs:
    - name: tail
      path: /var/log/auth.log
      tag: host.auth
      read_from_head: false
    - name: tail
      path: /var/log/app/security.log
      tag: app.security
      parser: json
  filters:
    - name: record_modifier
      match: '*'
      record:
        - host ${HOSTNAME}
        - env production
  outputs:
    - name: forward
      match: '*'
      host: logs.internal
      port: 24224
      tls: on
      tls.verify: on

Retention is a policy decision with a floor: keep security-relevant logs for at least a year, because intrusions are often discovered months after they begin, and regulators and insurers increasingly require it. Tier the storage — hot for thirty to ninety days of fast search, cold object storage beyond — and control who can read the archive, since logs contain personal data. Test restoration from cold storage before you need it.

Detections: rules that catch behaviour

A detection is a rule that turns events into an alert with a known meaning. Good detections describe attacker behaviour, not just anomalies: a login followed by MFA enrolment from a new country, a service account used from an IP it has never used, curl or python -c executed inside a container that never runs shells, an IAM policy change outside the deploy pipeline, a user reading a hundred times more records than their baseline. Map each detection to the technique it covers (the MITRE ATT&CK catalogue is the shared vocabulary) so that you can see which stages of an attack you would notice and which you would not.

title: Interactive shell spawned inside a container
id: 1c7a2e8f-3b4d-4c5e-9f0a-1b2c3d4e5f60
status: stable
description: A shell started in a production container; application containers never run one.
references:
  - internal runbook DET-014
tags:
  - attack.execution
  - attack.t1059.004
logsource:
  product: linux
  service: auditd
detection:
  selection:
    type: EXECVE
    a0|endswith:
      - '/sh'
      - '/bash'
      - '/dash'
    container_id|contains: '*'
  filter_debug:
    container_name|startswith: 'debug-'
  condition: selection and not filter_debug
falsepositives:
  - Engineers using an approved debug container (excluded by name)
level: high

Start with a small set of high-value detections rather than a vendor's thousand-rule pack: authentication anomalies, privilege changes, new persistence (new users, new keys, new cron or systemd units), execution in places it should not happen, unusual data access volumes, and known-bad indicators. Each rule ships with a runbook: what it means, how to confirm, what to do. A detection without a runbook produces an alert nobody can act on.

Alert fatigue is a security vulnerability

An alert queue that grows faster than it is worked is equivalent to no detection at all; the real alert is in there, unread. Treat alert volume like an SLO: every alert is investigated within its target time, and if the rate makes that impossible, the fix is tuning or automation, not a bigger queue. Measure per rule: how many alerts, how many true positives, median time to close. Rules with near-zero true positives get tuned, demoted to a dashboard, or deleted.

• Severity means something: critical pages a human; high is worked the same day; medium and below aggregate into a daily review.
• Enrich before alerting: attach the user's manager, the asset's owner, whether the IP is a known VPN egress, recent related alerts. Investigation time drops sharply.
• Automate the first step: an alert for a leaked key can revoke it; an alert for a new admin can post to the owner for confirmation. Humans handle the judgement, not the clicking.
• Suppress with expiry: a known false positive is excluded with a rule that names the reason and a date to reconsider.
• Close the loop: every incident retrospective asks which detection fired, which should have, and what to add.

The SRE track's alerting module makes the same argument for reliability alerts: alert on what matters, make every page actionable, and treat noise as a bug. Security alerting borrows all of it.

Test the detections

Detections rot silently: a log source stops shipping, a field is renamed, a parser breaks, a rule's threshold no longer matches the environment. Test them like code. Unit-test rule logic against sample events. Monitor log pipelines for volume drops per source, which is how you notice a broken agent or an attacker disabling logging. Most importantly, run the attack: adversary emulation tools (Atomic Red Team, Stratus Red Team for cloud) execute safe versions of real techniques, and a detection that does not fire is a finding for the detection engineering backlog.

stratus warmup aws.persistence.iam-create-backdoor-user
stratus detonate aws.persistence.iam-create-backdoor-user
# ...check that your "new IAM user created outside the pipeline" detection produced an alert...
stratus cleanup aws.persistence.iam-create-backdoor-user

Keep a coverage map: techniques on one axis, your detections on the other, with a colour for tested-and-fires, exists-but-untested, and absent. It shows leadership what the team can and cannot see, guides what to write next, and is the honest answer to the question every board eventually asks after a breach elsewhere: would we have noticed?