Secure a web service end to end

A one-page model of the service as it actually runs, and a ranked register of eight threats whose controls become the work plan.

1. 1.1
   Create the branch and the security directory. Draw the data flow diagram from the deployed reality: internet → EC2 (port 80, Docker) → app container; GitHub Actions → GHCR → the host pulls; you → SSM Run Command; app → Secrets Manager (soon). Mark the trust boundaries: internet edge, host to container, pipeline to host.

   bashCopy

   git switch -c feat/security
   mkdir -p security/controls security/detections security/incidents tests/security

2. 1.2
   List the assets in rank order and the entry points with their authentication. Be honest: today the API has no authentication and the host is reachable on port 80 from anywhere.

   textCopy

   # security/THREAT_MODEL.md (part 1)
   
   ASSETS (ranked)
     1. ability to run code on the host (leads to the AWS role and everything below)
     2. the instance's IAM role credentials (Secrets Manager read, SSM)
     3. Grafana admin password; the GHCR package; the repo's deploy role trust
     4. availability of the API
   
   ENTRY POINTS
     E1  HTTP :80 -> app         internet -> host -> container        (auth: none)
     E2  :3000 Grafana, :9090    my_ip only                           (auth: Grafana password)
     E3  SSM Run Command         AWS IAM (github role, my user)       (auth: OIDC / MFA)
     E4  image pull from GHCR    host -> internet                     (auth: none, public package)
     E5  GitHub Actions          repo push, PRs                       (auth: GitHub; OIDC to AWS)
     E6  instance metadata       any process on the host              (auth: IMDSv2 token if required)
   
   BOUNDARIES
     internet | security group | host (Amazon Linux, Docker) | containers
     GitHub | GHCR | host      and      GitHub OIDC | AWS role

3. 1.3
   Walk STRIDE over each entry point and rate the results. Keep the top eight; these are the controls you will build in the next phases. Each has a response, a specific control, and the name of the test that proves it.

   textCopy

   # security/THREAT_MODEL.md (part 2: register)
   
   T-01 E1 Tampering/Elevation: SSRF or RCE in the app reaches the metadata service and steals role creds
        HIGH x HIGH -> MITIGATE: IMDSv2 required (hop limit 1); container has no host network
        test: test_infra_policy::test_imdsv2_required
   T-02 E1 Elevation: container escape to host (root in container, docker socket, privileged)
        MED x HIGH  -> MITIGATE: non-root distroless image, no socket mounts, read-only rootfs
        test: test_image::test_runs_as_non_root, test_image::test_no_capabilities_needed
   T-03 E4 Tampering: a tampered or unsigned image is pulled and run
        MED x HIGH  -> MITIGATE: cosign keyless signing in release; host verifies before pull
        test: test_release_workflow::test_signature_verified_before_deploy
   T-04 E5 Info disclosure: a secret committed to the repo or printed in CI
        HIGH x HIGH -> MITIGATE: gitleaks gate; no secrets in env files; Secrets Manager at runtime
        test: test_image::test_no_secret_shaped_env, CI gate job
   T-05 E1 Info disclosure: vulnerable dependency or base image exploited
        MED x MED   -> MITIGATE: pip-audit + trivy gates; weekly rebuild
        test: test_image::test_no_fixable_critical_cves
   T-06 E3 Elevation: the GitHub deploy role is over-privileged (EC2FullAccess, SSMFullAccess)
        MED x HIGH  -> MITIGATE: least-privilege policy scoped to the instance and the state bucket
        test: test_infra_policy::test_deploy_role_no_wildcards
   T-07 host Repudiation: no record of commands run on the host or of logins
        MED x MED   -> MITIGATE: auditd + auth log shipped to CloudWatch; alarms
        test: detection drill D-01, D-02
   T-08 E1 DoS: unbounded requests exhaust the single host
        MED x MED   -> ACCEPT for this project (single host by design); note: WAF + ALB in the ECS project

   Check: Every threat has an entry point, a rating, a response, a control and a test name. Commit the file.

The EC2 instance requires IMDSv2, has no SSH path, runs a host firewall and auditd, patches itself, and ships its logs — all from Terraform and the boot script.

1. 2.1
   Require IMDSv2 on the instance and give the role permission to read one secret and to write logs. Replace the instance block's relevant parts in infra/main.tf.

   hclCopy

   resource "aws_instance" "app" {
     # ... existing arguments ...
   
     metadata_options {
       http_endpoint               = "enabled"
       http_tokens                 = "required"   # IMDSv2 only: a plain GET from an SSRF gets nothing
       http_put_response_hop_limit = 1            # tokens cannot be fetched from inside a container
     }
   
     root_block_device {
       volume_size = 16
       encrypted   = true
     }
   }
   
   resource "aws_iam_role_policy" "instance_least_privilege" {
     name = "${var.name}-instance"
     role = aws_iam_role.instance.id
     policy = jsonencode({
       Version = "2012-10-17"
       Statement = [
         {
           Sid      = "ReadOwnSecret"
           Effect   = "Allow"
           Action   = ["secretsmanager:GetSecretValue"]
           Resource = aws_secretsmanager_secret.app.arn
         },
         {
           Sid      = "ShipLogs"
           Effect   = "Allow"
           Action   = ["logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams"]
           Resource = "${aws_cloudwatch_log_group.host.arn}:*"
         }
       ]
     })
   }

   The AmazonSSMManagedInstanceCore attachment from the original project stays; it is what lets you run commands without SSH.
2. 2.2
   Create the secret that the app will read at runtime (the Grafana admin password moves here too), and the log group.

   hclCopy

   resource "aws_secretsmanager_secret" "app" {
     name                    = "${var.name}/app"
     recovery_window_in_days = 7
   }
   
   resource "aws_secretsmanager_secret_version" "app" {
     secret_id     = aws_secretsmanager_secret.app.id
     secret_string = jsonencode({
       grafana_password = var.grafana_password   # written once from your tfvars; rotate in the console later
       api_signing_key  = random_password.api_signing_key.result
     })
   }
   
   resource "random_password" "api_signing_key" {
     length  = 48
     special = false
   }
   
   resource "aws_cloudwatch_log_group" "host" {
     name              = "/${var.name}/host"
     retention_in_days = 365
   }

   Add random = { source = "hashicorp/random", version = "~> 3.6" } to required_providers. The secret's value ends up in Terraform state, which is why the state bucket is encrypted and private; for a production system, write the value out of band and only reference the secret here.
3. 2.3
   Extend the boot script with hardening: nftables default-deny with only 80 (and 3000/9090 from your IP via the security group, which stays), auditd with the rules from the hardening module, unattended security updates, and the CloudWatch agent shipping the auth log, the audit log and the app's security log.

   bashCopy

   # infra/user_data.sh (append after Docker is installed, before the compose up)
   dnf install -y nftables audit amazon-cloudwatch-agent dnf-automatic
   
   cat > /etc/nftables.conf <<'NFT'
   flush ruleset
   table inet filter {
     chain input {
       type filter hook input priority 0; policy drop;
       iif lo accept
       ct state established,related accept
       tcp dport { 80, 3000, 9090 } accept
       icmp type echo-request limit rate 5/second accept
     }
     chain forward { type filter hook forward priority 0; policy accept; }
     chain output  { type filter hook output priority 0; policy accept; }
   }
   NFT
   systemctl enable --now nftables
   
   cat > /etc/audit/rules.d/hardening.rules <<'AUD'
   -w /etc/passwd -p wa -k identity
   -w /etc/shadow -p wa -k identity
   -w /etc/sudoers -p wa -k privilege
   -w /etc/sudoers.d/ -p wa -k privilege
   -w /home/ -p wa -k home_changes
   -w /root/.ssh/ -p wa -k ssh_keys
   -a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=unset -k root_commands
   -e 2
   AUD
   augenrules --load && systemctl enable --now auditd
   
   sed -i 's/^apply_updates = .*/apply_updates = yes/; s/^upgrade_type = .*/upgrade_type = security/' /etc/dnf/automatic.conf
   systemctl enable --now dnf-automatic.timer
   
   cat > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json <<'CW'
   {
     "logs": {
       "logs_collected": {
         "files": {
           "collect_list": [
             {"file_path": "/var/log/secure",      "log_group_name": "/${name}/host", "log_stream_name": "{instance_id}/auth"},
             {"file_path": "/var/log/audit/audit.log", "log_group_name": "/${name}/host", "log_stream_name": "{instance_id}/audit"},
             {"file_path": "/opt/app/logs/security.log", "log_group_name": "/${name}/host", "log_stream_name": "{instance_id}/app-security"}
           ]
         }
       }
     }
   }
   CW
   systemctl enable --now amazon-cloudwatch-agent

   ${name} is filled by Terraform's templatefile; add name = var.name to the template variables in main.tf. Docker manages the forward chain, so it stays at accept; the security group still limits 3000/9090 to your IP.
4. 2.4
   Make the app write its security events to a file the agent ships. Add the security_event helper from the logging module to app/main.py, log authz.denied and auth.signature.invalid, and mount /opt/app/logs into the container in deploy/compose.yml.

   yamlCopy

   # deploy/compose.yml (app service additions)
       volumes:
         - /opt/app/logs:/logs
       environment:
         SECURITY_LOG: /logs/security.log
       read_only: true
       tmpfs: ["/tmp"]
       security_opt: ["no-new-privileges:true"]
       cap_drop: ["ALL"]

   read_only plus a tmpfs for /tmp gives the container an immutable filesystem; no-new-privileges and cap_drop: ALL are the Compose equivalents of the Kubernetes restricted profile.
5. 2.5
   Write the infrastructure policy tests: they run terraform plan -json and assert the settings exist, so a future edit that drops IMDSv2 or widens the role fails CI.

   pythonCopy

   # tests/security/test_infra_policy.py
   import json
   import subprocess
   
   import pytest
   
   
   @pytest.fixture(scope="module")
   def plan():
       subprocess.run(["terraform", "-chdir=infra", "init", "-input=false", "-backend=false"], check=True, capture_output=True)
       subprocess.run(["terraform", "-chdir=infra", "plan", "-input=false", "-out=tfplan",
                       "-var=image=ghcr.io/example/x:latest", "-var=grafana_password=test", "-var=my_ip=203.0.113.9/32"],
                      check=True, capture_output=True)
       out = subprocess.run(["terraform", "-chdir=infra", "show", "-json", "tfplan"], check=True, capture_output=True, text=True).stdout
       return json.loads(out)
   
   
   def resources(plan, rtype):
       return [r for r in plan["planned_values"]["root_module"]["resources"] if r["type"] == rtype]
   
   
   def test_imdsv2_required(plan):
       inst = resources(plan, "aws_instance")[0]["values"]
       assert inst["metadata_options"][0]["http_tokens"] == "required"
       assert inst["metadata_options"][0]["http_put_response_hop_limit"] == 1
   
   
   def test_root_volume_encrypted(plan):
       inst = resources(plan, "aws_instance")[0]["values"]
       assert inst["root_block_device"][0]["encrypted"] is True
   
   
   def test_no_ssh_from_internet(plan):
       for sg in resources(plan, "aws_security_group"):
           for rule in sg["values"].get("ingress", []):
               if rule["from_port"] <= 22 <= rule["to_port"]:
                   assert "0.0.0.0/0" not in rule["cidr_blocks"]
   
   
   def test_instance_policy_has_no_wildcard_actions(plan):
       for pol in resources(plan, "aws_iam_role_policy"):
           doc = json.loads(pol["values"]["policy"])
           for st in doc["Statement"]:
               actions = st["Action"] if isinstance(st["Action"], list) else [st["Action"]]
               assert "*" not in actions and not any(a.endswith(":*") for a in actions)

   -backend=false lets the plan run in CI without touching the real state bucket; the plan is for assertions only.
6. 2.6
   Apply the infrastructure changes from your laptop (the pipeline will own applies again after this phase), wait for the instance to replace itself (user data changed), and confirm the hardening took effect through Run Command.

   bashCopy

   cd infra && terraform init && terraform apply && cd ..
   INSTANCE=$(aws ec2 describe-instances --filters Name=tag:Name,Values=zero-to-prod Name=instance-state-name,Values=running --query 'Reservations[].Instances[].InstanceId' --output text)
   aws ssm send-command --instance-ids "$INSTANCE" --document-name AWS-RunShellScript \
     --parameters 'commands=["nft list ruleset | head -20","auditctl -s","systemctl is-active amazon-cloudwatch-agent dnf-automatic.timer","curl -s -m 2 http://169.254.169.254/latest/meta-data/ || echo IMDSv1-blocked"]' \
     --query Command.CommandId --output text

   Check: The command output (Systems Manager → Run Command → the command → the instance) shows the drop policy, enabled 2 for auditd, both services active, and IMDSv1-blocked. In CloudWatch Logs, the /zero-to-prod/host group has auth and audit streams receiving lines.

A distroless non-root image pinned by digest, secrets fetched at runtime, and gates that block secrets, vulnerable dependencies, bad infrastructure and unsigned images.

1. 3.1
   Rewrite the Dockerfile to a distroless, non-root final stage pinned by digest. Get the current digests with docker buildx imagetools inspect python:3.12-slim and ... gcr.io/distroless/python3-debian12:nonroot.

   dockerfileCopy

   # syntax=docker/dockerfile:1
   FROM python:3.12-slim@sha256:REPLACE_WITH_CURRENT_DIGEST AS build
   WORKDIR /src
   COPY requirements.txt .
   RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
   
   FROM gcr.io/distroless/python3-debian12:nonroot@sha256:REPLACE_WITH_CURRENT_DIGEST
   COPY --from=build /install /usr/local
   COPY app /app/app
   WORKDIR /app
   ENV PYTHONPATH=/usr/local/lib/python3.12/site-packages PYTHONUNBUFFERED=1
   USER nonroot:nonroot
   EXPOSE 8000
   ENTRYPOINT ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

   Distroless has no shell, so the previous HEALTHCHECK with python -c still works but a curl-based one would not. Confirm the site-packages path by running docker run --rm --entrypoint python IMAGE -c 'import sys; print(sys.path)'.
2. 3.2
   Fetch the secret at runtime with the instance role instead of the .env file. The Grafana password stays in Compose for now (it is not the app's secret); the app reads its own signing key from Secrets Manager on startup and uses it to verify a request signature on a new protected endpoint — giving the API its first real authentication.

   pythonCopy

   # app/secrets.py
   import json
   import os
   from functools import lru_cache
   
   import boto3
   
   
   @lru_cache(maxsize=1)
   def app_secrets() -> dict:
       name = os.environ.get("APP_SECRET_NAME", "zero-to-prod/app")
       client = boto3.client("secretsmanager", region_name=os.environ.get("AWS_REGION", "eu-west-1"))
       return json.loads(client.get_secret_value(SecretId=name)["SecretString"])
   
   
   # app/main.py additions
   import hmac
   import hashlib
   
   from fastapi import Header, HTTPException, Request
   
   from app.secrets import app_secrets
   
   
   @app.post("/admin/reindex")
   async def reindex(request: Request, x_signature: str = Header(default="")):
       body = await request.body()
       expected = hmac.new(app_secrets()["api_signing_key"].encode(), body, hashlib.sha256).hexdigest()
       if not hmac.compare_digest(expected, x_signature):
           security_event("auth.signature.invalid", actor=request.client.host, target="/admin/reindex", outcome="denied", request_id=request.headers.get("x-request-id", "-"))
           raise HTTPException(status_code=404)
       return {"status": "queued"}

   Add boto3 to requirements.txt. The container reaches Secrets Manager through the instance role via IMDSv2 — which works from inside the container only because the hop limit is 1 and Docker's default bridge is one hop; if you later move to a different network mode, use the container credential provider instead.
3. 3.3
   Add the image tests. The CVE test uses Trivy with --ignore-unfixed so unfixable base-image findings do not block, while fixable criticals do.

   pythonCopy

   # tests/security/test_image.py
   import json
   import os
   import subprocess
   
   IMAGE = os.environ.get("IMAGE_UNDER_TEST", "zero-to-prod:test")
   
   
   def inspect():
       out = subprocess.run(["docker", "image", "inspect", IMAGE], capture_output=True, text=True, check=True).stdout
       return json.loads(out)[0]["Config"]
   
   
   def test_runs_as_non_root():
       assert inspect().get("User", "") not in ("", "0", "root")
   
   
   def test_no_secret_shaped_env():
       env = inspect().get("Env") or []
       assert not [e for e in env if any(k in e.upper() for k in ("PASSWORD", "SECRET", "TOKEN", "AKIA"))]
   
   
   def test_no_fixable_critical_cves():
       r = subprocess.run(["trivy", "image", "--quiet", "--severity", "CRITICAL", "--ignore-unfixed", "--exit-code", "1", IMAGE],
                          capture_output=True, text=True)
       assert r.returncode == 0, r.stdout
   
   
   def test_no_secrets_in_layers():
       r = subprocess.run(["trivy", "image", "--quiet", "--scanners", "secret", "--exit-code", "1", IMAGE], capture_output=True, text=True)
       assert r.returncode == 0, r.stdout

4. 3.4
   Add the two-tier security workflow from the DevSecOps module as .github/workflows/security.yml: a gate job (gitleaks, Semgrep OWASP rules at ERROR, pip-audit strict, Checkov on infra/, then build the image and run tests/security/test_image.py) and a report job (full Semgrep to SARIF, never blocking). Install Trivy in the job with aquasecurity/setup-trivy@v0.2.3 and Terraform with hashicorp/setup-terraform@v3 for the policy tests.

   The exact YAML is in the DevSecOps lesson; the additions here are the image build (docker build -t zero-to-prod:test .) and pytest tests/security -q after the scanners.
5. 3.5
   Sign the image in release.yml right after the push, and make the host verify the signature before it deploys. Add to the build job (after docker/build-push-action):

   yamlCopy

         - uses: sigstore/cosign-installer@v3
         - name: Sign the image by digest (keyless, this workflow's identity)
           run: cosign sign --yes "${IMAGE}@${DIGEST}"
           env:
             IMAGE: ${{ env.IMAGE }}
             DIGEST: ${{ steps.build.outputs.digest }}

   docker/build-push-action exposes the digest as steps.<id>.outputs.digest; give the build step id: build. The job already has id-token: write.
6. 3.6
   Change the deploy job so the host verifies before pulling: install cosign on the host in user_data.sh (download the release binary and chmod +x), and change the Run Command to verify the exact digest against your workflow identity, aborting if verification fails.

   bashCopy

   # the commands sent through SSM by the deploy job (one line each in the JSON parameters)
   cd /opt/app
   cosign verify --certificate-identity-regexp '^https://github.com/YOUR_GITHUB_USER/zero-to-prod/.github/workflows/release.yml@refs/heads/main$' --certificate-oidc-issuer https://token.actions.githubusercontent.com ghcr.io/YOUR_GITHUB_USER/zero-to-prod@DIGEST || exit 1
   sed -i 's#^APP_IMAGE=.*#APP_IMAGE=ghcr.io/YOUR_GITHUB_USER/zero-to-prod@DIGEST#' .env
   docker compose pull app && docker compose up -d app

   Deploy by digest, not tag: the workflow passes ${{ steps.build.outputs.digest }} into the command. A signature check by tag would be meaningless because tags move.
7. 3.7
   Open a pull request, watch the gates run, merge, and watch the release sign the image and the host verify it. Then prove the gate with a negative test: push a branch with a fake AKIA... key in a file and confirm the pull request is blocked.

   bashCopy

   git add . && git commit -m "feat: hardened image, runtime secrets, scanning gates, signed deploys"
   git push -u origin feat/security
   gh pr create --fill && gh pr checks --watch
   gh pr merge --squash --delete-branch && git switch main && git pull && gh run watch

   Check: The deploy job's Run Command output includes Verified OK from cosign before the pull. A test branch containing a secret shows a red Secrets (blocks) step.

Three detections on the shipped logs that alarm within minutes, each with a runbook line, implemented as code in Terraform.

1. 4.1
   Define the three detections as CloudWatch metric filters and alarms in infra/logging.tf: an SSH authorized_keys change (persistence), a burst of root commands outside the deploy window (privilege use), and repeated invalid signatures on the admin endpoint (credential guessing).

   hclCopy

   locals {
     detections = {
       ssh_key_change = {
         pattern   = "[..., key=ssh_keys]"              # auditd lines carrying the ssh_keys rule key
         threshold = 1
         summary   = "authorized_keys modified on the host"
       }
       root_command_burst = {
         pattern   = "[..., key=root_commands]"
         threshold = 20                                 # a deploy runs a handful; a human shell runs many
         summary   = "unusual volume of root commands"
       }
       admin_signature_failures = {
         pattern   = "{ $.event = \"auth.signature.invalid\" }"
         threshold = 10
         summary   = "repeated invalid signatures on /admin"
       }
     }
   }
   
   resource "aws_cloudwatch_log_metric_filter" "det" {
     for_each       = local.detections
     name           = "${var.name}-${each.key}"
     log_group_name = aws_cloudwatch_log_group.host.name
     pattern        = each.value.pattern
     metric_transformation {
       name      = each.key
       namespace = "${var.name}/security"
       value     = "1"
     }
   }
   
   resource "aws_cloudwatch_metric_alarm" "det" {
     for_each            = local.detections
     alarm_name          = "${var.name}-${each.key}"
     alarm_description   = "${each.value.summary} — runbook: security/detections/${each.key}.md"
     namespace           = "${var.name}/security"
     metric_name         = each.key
     statistic           = "Sum"
     period              = 300
     evaluation_periods  = 1
     threshold           = each.value.threshold
     comparison_operator = "GreaterThanOrEqualToThreshold"
     treat_missing_data  = "notBreaching"
     alarm_actions       = [aws_sns_topic.security.arn]
   }
   
   resource "aws_sns_topic" "security" {
     name = "${var.name}-security-alerts"
   }
   
   resource "aws_sns_topic_subscription" "email" {
     topic_arn = aws_sns_topic.security.arn
     protocol  = "email"
     endpoint  = var.security_email
   }

   Add variable "security_email" and set it in tfvars; confirm the subscription from the email AWS sends. The audit log lines are space-delimited, hence the [...] pattern syntax; the app log is JSON, hence the { $.event = ... } form.
2. 4.2
   Write a runbook per detection in security/detections/<name>.md: what it means, how to confirm (the CloudWatch Logs Insights query), and the containment step (isolate the instance by swapping its security group for one with no rules, snapshot, then investigate via Run Command).

   textCopy

   # security/detections/ssh_key_change.md
   
   **Meaning:** a file under /root/.ssh or /home/*/.ssh changed. Nobody should be adding SSH keys; SSH is not
   an access path on this host. Treat as persistence until proven otherwise.
   
   **Confirm (Logs Insights on /zero-to-prod/host):**
     fields @timestamp, @message | filter @message like /key=ssh_keys/ | sort @timestamp desc | limit 50
   Look at the auid (which login) and the exe (which process) in the audit line.
   
   **Contain:** if not you and not a deploy:
     aws ec2 modify-instance-attribute --instance-id I --groups SG_QUARANTINE   # no rules: isolated
     aws ec2 create-snapshot --volume-id V --description "evidence $(date -u +%FT%TZ)"
   Then Run Command: cat the changed authorized_keys, last -F, ps -eo pid,user,lstart,cmd.
   
   **Recover:** rebuild the instance from Terraform (user_data recreates a clean host); rotate the
   instance role's secrets; add the finding to security/incidents/.

   Create the quarantine security group (aws_security_group.quarantine with no ingress and no egress) in Terraform so it exists before you need it.
3. 4.3
   Apply, then confirm the metric filters parse real lines by running a Logs Insights query for each pattern against the last hour.

   bashCopy

   cd infra && terraform apply && cd ..
   aws logs start-query --log-group-name /zero-to-prod/host --start-time $(($(date +%s)-3600)) --end-time $(date +%s) \
     --query-string 'fields @timestamp, @message | filter @message like /key=root_commands/ | sort @timestamp desc | limit 5' \
     --query queryId --output text

   Check: aws logs get-query-results --query-id ID returns audit lines from the deploy's docker compose commands, proving the pipeline from auditd to CloudWatch works. The three alarms show OK (or INSUFFICIENT_DATA until the first datapoint) in the CloudWatch console.

Each detection proven by triggering its behaviour, one control removed and caught by a test, and a write-up that links every claim to evidence.

1. 5.1
   Drill D-01: plant an SSH key through Run Command (the only way in, which is itself worth noting), then time the alarm.

   bashCopy

   date -u
   aws ssm send-command --targets Key=tag:Name,Values=zero-to-prod --document-name AWS-RunShellScript \
     --parameters 'commands=["mkdir -p /root/.ssh","echo ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDRILLKEYDRILLKEYDRILLKEYDRILLKEYDRILLKEY drill >> /root/.ssh/authorized_keys"]' \
     --query Command.CommandId --output text

   Check: Within about five minutes (one alarm period) the zero-to-prod-ssh_key_change alarm goes to ALARM and the email arrives with the runbook path. Record injection time and alarm time.
2. 5.2
   Follow the runbook: run the Logs Insights query, identify the audit line (it shows the SSM agent's session as the actor), and remove the key. Then rebuild the host from Terraform as the runbook says, because a real incident would not trust the cleaned host.

   bashCopy

   aws ssm send-command --targets Key=tag:Name,Values=zero-to-prod --document-name AWS-RunShellScript \
     --parameters 'commands=["sed -i /drill/d /root/.ssh/authorized_keys"]' --query Command.CommandId --output text
   cd infra && terraform apply -replace=aws_instance.app && cd ..

   Check: The alarm returns to OK after the next clean period; a fresh instance comes up with the hardening applied automatically.
3. 5.3
   Drill D-03: hammer the admin endpoint with bad signatures from your laptop and confirm the application-log detection fires.

   bashCopy

   IP=$(cd infra && terraform output -raw public_ip)
   for i in $(seq 1 15); do curl -s -o /dev/null -X POST "http://$IP/admin/reindex" -H 'X-Signature: nope' -d '{}'; done

   Check: The admin_signature_failures alarm fires within a period; the app-security stream shows fifteen auth.signature.invalid events with your IP as the actor.
4. 5.4
   Drill D-04 (the leaked-key simulation): create a test IAM user with a key, use it once from your laptop, then "discover the leak" and run the secrets module's response in order — revoke, audit with CloudTrail lookup by access key id, contain, write up. Delete the user afterwards.

   bashCopy

   aws iam create-user --user-name drill-leak && aws iam create-access-key --user-name drill-leak > /tmp/drillkey.json
   KEY=$(jq -r .AccessKey.AccessKeyId /tmp/drillkey.json)
   AWS_ACCESS_KEY_ID=$KEY AWS_SECRET_ACCESS_KEY=$(jq -r .AccessKey.SecretAccessKey /tmp/drillkey.json) aws sts get-caller-identity
   # response:
   aws iam update-access-key --user-name drill-leak --access-key-id "$KEY" --status Inactive      # 1. revoke
   aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue="$KEY" --max-results 20   # 2. audit
   aws iam delete-access-key --user-name drill-leak --access-key-id "$KEY" && aws iam delete-user --user-name drill-leak
   rm /tmp/drillkey.json

   Check: CloudTrail shows the GetCallerIdentity call from your IP under that key; the timeline in the write-up records revoke-before-audit.
5. 5.5
   Prove the tests hold the line: in a branch, set http_tokens = "optional" and change the Dockerfile's USER to root, open a pull request, and confirm both test_imdsv2_required and test_runs_as_non_root fail. Close the pull request.

   Check: Two red tests naming the exact controls; the merge is blocked.
6. 5.6
   Write the control records and the incident write-up, and finish the README section.

   textCopy

   # security/controls/C-01-imdsv2.md
   Control:   instance metadata requires IMDSv2 with hop limit 1
   Threat:    T-01 (SSRF/RCE to role credentials)
   Evidence:  infra/main.tf metadata_options; tests/security/test_infra_policy.py::test_imdsv2_required (CI, every PR)
   Verified:  2026-09-21 by Run Command: plain GET to 169.254.169.254 returns nothing
   Owner:     you
   
   # security/incidents/2026-09-21-drill.md
   D-01 ssh key planted 10:02Z -> alarm 10:06Z -> contained 10:14Z -> host rebuilt 10:31Z   (detect 4 min, contain 12 min)
   D-03 15 bad signatures 10:40Z -> alarm 10:45Z
   D-04 leaked key used 10:50Z -> revoked 10:52Z -> audited 10:55Z
   Improvement shipped: added /home/*/.ssh to the audit watch list (was /root only) after D-01 review.

   One record per threat in the register; the write-up lists detect and contain times and at least one improvement you actually made.
7. 5.7
   Commit and merge through a pull request. The service now has a threat model, eight controls with tests, gates in the pipeline, signed deploys, shipped logs, three tested detections and a recorded response — the full loop from the Cybersecurity capstone, on a real host.

   bashCopy

   git add security tests infra app deploy .github Dockerfile
   git commit -m "security: detections, drills, control records and evidence"
   git push -u origin HEAD && gh pr create --fill && gh pr merge --squash --delete-branch
   git switch main && git pull

   When you are done experimenting, terraform destroy as in the original project; the SNS subscription and log group cost cents while they exist, and the secret has a 7-day recovery window before it is fully removed.

The container cannot reach Secrets Manager (credentials error)

With IMDSv2 hop limit 1, the container gets credentials through the bridge network's single hop. If you set network_mode: host or a custom network with extra hops, raise the hop limit to 2 or use the ECS/EC2 container credential provider. Check curl -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' from inside the container via Run Command.

The distroless image fails with ModuleNotFoundError

The --prefix=/install layout puts packages under /install/lib/python3.12/site-packages; after COPY --from=build /install /usr/local, set PYTHONPATH=/usr/local/lib/python3.12/site-packages. Verify with docker run --rm --entrypoint python IMAGE -c 'import fastapi'.

cosign verify on the host fails with a certificate identity mismatch

The identity regexp must match the workflow file path and ref exactly: .github/workflows/release.yml@refs/heads/main. Print the signature's certificate with cosign verify ... --output text to see the actual identity.

The audit metric filter never matches

Audit lines look like type=SYSCALL ... key="ssh_keys" with quotes; the space-delimited pattern [..., key=ssh_keys] may need to be [..., key="ssh_keys"] depending on the agent's formatting. Test patterns in the console's filter tester against real lines from the stream.

The root_command_burst alarm fires on every deploy

Deploys run docker compose as root through SSM. Raise the threshold above a deploy's command count, or exclude the SSM agent's session by adding -F exe!=/usr/bin/ssm-session-worker to the audit rule — but note that exclusion also hides an attacker using SSM.

Terraform wants to recreate the instance every apply

user_data_replace_on_change = true plus a template that includes a changing value (a timestamp, a random) causes this. Keep the template deterministic; only real changes should replace the host.

test_infra_policy fails in CI with a provider download error

The policy tests run terraform init -backend=false, which still downloads the AWS provider; ensure the CI job has network access and hashicorp/setup-terraform installed a 1.10+ binary.

• Put the service behind an ALB with WAF in the Terraform an AWS environment project, which retires threat T-08 (DoS on a single host) that this project accepted.
• Replace metric filters with a proper detection pipeline: ship the same logs to an OpenSearch or Loki stack and write the Sigma rules from the detection module.
• Run the Signed and attested builds project to add SBOM attestations and SLSA provenance on top of the signature you added here.
• Add the SRE track's SLO alerts to the same host, and notice that the security alarms and the reliability alerts want the same routing and runbook discipline.
• Do the Build a detection lab project to practise the response side with a full SIEM and adversary emulation.