Education › Cybersecurity › Guided project

Signed and attested builds

Answer three questions about the container you are about to run in production: what is inside it, who built it, and can I prove both? You will generate an SBOM on every build and scan it against vulnerability databases, sign the image keylessly with Sigstore so there is no private key to leak, attach SLSA build provenance that ties the image to a commit and a workflow run, harden the build pipeline itself against the attacks that make all of this necessary, and then put a verification gate in front of the deploy that refuses anything unsigned. You finish by attacking your own pipeline — a tampered image, a backdoored dependency, a workflow run from the wrong branch — and watching each attempt fail at a specific control you can name.

Intermediate about 5 hours 7 phases · 35 steps 0 / 35 done
Overview 1. Know what you ship 2. Sign the image, keylessly 3. Provenance: prove which build made it 4. Harden the build itself 5. The gate: refuse what you cannot verify 6. Attack your own pipeline 7. Optional: let the cluster enforce it Troubleshooting
What you will have at the end

The zero-to-production repository builds images that carry a CycloneDX SBOM attestation, a Sigstore keyless signature and GitHub-issued SLSA v1 provenance, all verifiable by anyone with cosign verify and gh attestation verify; a verify-image.sh gate that the deploy runs first and that refuses an image whose signature, provenance repository or workflow identity does not match; every third-party action pinned to a commit SHA with least-privilege tokens; an OSV scan that fails the build on a fixable critical; and a security/SUPPLY-CHAIN.md recording four attacks you ran against your own pipeline and which control stopped each one.

Before you start
  • The zero-to-prod project: the repository, the GHCR image, and the release workflow that builds it
  • The security track's modules on the software supply chain, containers and DevSecOps
  • Docker, the GitHub CLI (gh 2.60+ for gh attestation), and a GHCR package you own
  • Optional: the GitOps with Argo CD project, if you want the Kubernetes admission-control path at the end
Tools you will install
  • Syft — generates the SBOM from the image — every package, version and licence that ends up in the layers ↗
  • Cosign (Sigstore) — keyless signing and attestation using a short-lived certificate tied to the workflow's OIDC identity ↗
  • GitHub artifact attestations — SLSA v1 build provenance issued by GitHub, verified with gh attestation verify ↗
  • OSV-Scanner — scans the SBOM against the OSV database; knows which findings have fixes ↗
  • Trivy — a second opinion on image CVEs and a familiar name in interviews; used for the fail-on-fixable gate ↗
  • Kyverno — optional final phase: the cluster refuses images that are not signed by your workflow ↗
Repository layout at the end
zero-to-prod/
├── .github/workflows/
│   ├── release.yml            # build → SBOM → sign → attest → provenance
│   └── supply-chain.yml       # dependency review, OSV scan, pinned-action check
├── security/
│   ├── SUPPLY-CHAIN.md        # what is produced, how to verify, the four attacks
│   ├── verify-image.sh        # the deploy gate: signature + provenance + identity
│   └── policy/
│       ├── kyverno-verify-images.yaml   # optional cluster enforcement
│       └── allowed-identities.env       # repo, workflow ref, issuer
├── deploy/
│   └── pull-and-run.sh        # calls verify-image.sh before docker compose up
└── .github/dependabot.yml

Tick each step as you finish it — your progress is saved in this browser only (back up or restore on the hub). Every code block has a copy button. If something goes wrong, the troubleshooting section at the end covers the usual suspects.

Phase 1

Know what you ship

Dependencies pinned and watched, an SBOM generated from the built image on every run, and a vulnerability scan that fails the build only on things you can actually fix.

  1. Start in the zero-to-prod repository on a branch, and look at what you are currently shipping. docker history shows the layers; the image has dozens of OS packages you never chose, which is exactly why an SBOM matters.
    bash
    cd zero-to-prod && git switch -c feat/supply-chain
docker pull ghcr.io/$(gh api user -q .login)/zero-to-prod:latest
docker history --no-trunc ghcr.io/$(gh api user -q .login)/zero-to-prod:latest | head -12
  2. Install Syft and generate an SBOM locally in CycloneDX JSON. Count the components: the Python packages you declared are a small fraction of the total.
    bash
    brew install syft   # or: curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin
IMAGE=ghcr.io/$(gh api user -q .login)/zero-to-prod:latest
syft "$IMAGE" -o cyclonedx-json > sbom.cdx.json
python3 -c "import json; d=json.load(open('sbom.cdx.json')); c=d['components']; print(len(c), 'components'); \
import collections; print(collections.Counter(x.get('type','?') for x in c)); \
print([x['name'] for x in c if x.get('purl','').startswith('pkg:pypi')][:10])"
    Typical result for the FastAPI image: 100–150 components, of which a dozen are PyPI packages and the rest are Debian packages from the base image. If someone asks "are you affected by CVE-2026-xxxx in libexpat?", this file is the answer.
  3. Scan the SBOM with OSV-Scanner. Scanning the SBOM rather than the image means the scan is reproducible — the same file always gives the same answer, which matters when you are arguing about whether a build regressed.
    bash
    brew install osv-scanner   # or download from the releases page
osv-scanner scan source --sbom sbom.cdx.json --format table | head -30
osv-scanner scan source --sbom sbom.cdx.json --format json > osv.json
python3 -c "import json; d=json.load(open('osv.json')); \
print(sum(len(p.get('vulnerabilities',[])) for r in d.get('results',[]) for p in r.get('packages',[])), 'vulns')"
  4. Pin the Python dependencies with hashes so a compromised or re-uploaded package cannot change under you. uv writes a lock file with hashes; pip install --require-hashes refuses anything that does not match.
    bash
    uv pip compile pyproject.toml --generate-hashes -o requirements.lock
head -12 requirements.lock
# In the Dockerfile, replace the install line with the locked one:
#   COPY requirements.lock .
#   RUN python -m venv /venv && /venv/bin/pip install --require-hashes --no-deps -r requirements.lock
    --no-deps is deliberate: the lock file already lists the full closure, so pip must not resolve anything itself. If the build fails with "hashes are required", one dependency is missing from the lock — regenerate it rather than dropping the flag.
  5. Turn on Dependabot for both the Python dependencies and the GitHub Actions, so the pins you just made do not rot.
    yaml
    # .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: pip
    directory: "/"
    schedule: {interval: weekly}
    open-pull-requests-limit: 5
    groups:
      python-minor:
        update-types: [minor, patch]
  - package-ecosystem: github-actions
    directory: "/"
    schedule: {interval: weekly}
  - package-ecosystem: docker
    directory: "/"
    schedule: {interval: weekly}
  6. Add the supply-chain workflow: dependency review on pull requests (it blocks a PR that introduces a known-vulnerable package), and an OSV scan of the repository's lock file.
    yaml
    # .github/workflows/supply-chain.yml
name: supply-chain

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: "17 6 * * 1"        # Monday morning: new CVEs land against old code

permissions:
  contents: read

jobs:
  dependency-review:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/dependency-review-action@v4
        with:
          fail-on-severity: high
          comment-summary-in-pr: always

  osv:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: google/osv-scanner-action/osv-scanner-action@v2.6.0
        with:
          scan-args: |-
            --lockfile=requirements.lock
            --format=table
  7. Commit and open the pull request. Watch dependency review comment on it, then merge.
    bash
    git add .github Dockerfile requirements.lock pyproject.toml
git commit -m "supply-chain: hash-pinned deps, dependabot, dependency review and OSV scanning"
git push -u origin feat/supply-chain
gh pr create --fill && gh pr checks --watch
gh pr merge --squash --delete-branch && git switch main && git pull
Phase 2

Sign the image, keylessly

Every image pushed to GHCR carries a Sigstore signature made with a short-lived certificate bound to the workflow's OIDC identity — no key material anywhere — and an SBOM attestation attached to the same digest.

  1. Understand what keyless means before you use it. Cosign asks Fulcio for a certificate valid for about ten minutes, proving the holder of a particular OIDC identity — here repo:you/zero-to-prod:ref:refs/heads/main — signed this digest; the signature and certificate go into the Rekor transparency log. Nothing to store, nothing to leak, and a public record that the signature existed at a point in time.
    bash
    brew install cosign   # v3.x
cosign version
# Look at a signature someone else made, to see the shape of the thing:
cosign verify ghcr.io/sigstore/cosign/cosign:v3.1.3 \
  --certificate-identity-regexp 'https://github.com/sigstore/cosign/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  -o text 2>&1 | head -20
  2. In release.yml, capture the image digest from the build step — everything after this point signs the *digest*, never the tag. A tag can be moved; a digest is the content.
    yaml
    # .github/workflows/release.yml — the build job, with pinned actions
permissions:
  contents: read
  packages: write
  id-token: write          # required for keyless signing and for provenance
  attestations: write      # required for GitHub artifact attestations

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      digest: ${{ steps.push.outputs.digest }}
      image: ghcr.io/${{ github.repository }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=sha
            type=raw,value=latest,enable={{is_default_branch}}
      - id: push
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          provenance: false          # GitHub's attestation below replaces BuildKit's
          sbom: false                # we attach our own with cosign
  3. Add the signing job. It signs the digest, generates the SBOM from the pushed image, and attaches the SBOM as a CycloneDX attestation on the same digest. --yes skips the interactive confirmation, which is required in CI.
    yaml
      sign:
    needs: build
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write
    env:
      IMAGE: ${{ needs.build.outputs.image }}@${{ needs.build.outputs.digest }}
    steps:
      - uses: sigstore/cosign-installer@v4
        with:
          cosign-release: v3.1.3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Sign the image digest
        run: cosign sign --yes "$IMAGE"
      - name: Generate the SBOM from the pushed image
        uses: anchore/sbom-action@v0.24.2
        with:
          image: ${{ env.IMAGE }}
          format: cyclonedx-json
          output-file: sbom.cdx.json
          upload-artifact: false
      - name: Attach the SBOM as an attestation
        run: cosign attest --yes --type cyclonedx --predicate sbom.cdx.json "$IMAGE"
      - name: Fail on fixable critical vulnerabilities
        uses: aquasecurity/trivy-action@0.29.0
        with:
          image-ref: ${{ env.IMAGE }}
          severity: CRITICAL
          ignore-unfixed: true
          exit-code: "1"
          format: table
    ignore-unfixed: true is the difference between a gate people respect and one they disable. A critical with no available fix is a decision for a human and a record in the risk register, not a build failure at 2 a.m.
  4. Push, watch the workflow, then verify the signature from your laptop as any consumer would. Note that verification needs no credentials at all.
    bash
    git switch -c feat/sign-images && git add .github && git commit -m "release: keyless signing and SBOM attestation"
git push -u origin feat/sign-images && gh pr create --fill && gh pr merge --squash --delete-branch
git switch main && git pull && gh run watch
REPO=$(gh api user -q .login)/zero-to-prod
DIGEST=$(cosign triangulate --type digest ghcr.io/$REPO:latest 2>/dev/null || docker buildx imagetools inspect ghcr.io/$REPO:latest --format '{{.Manifest.Digest}}')
cosign verify "ghcr.io/$REPO@$DIGEST" \
  --certificate-identity-regexp "^https://github.com/$REPO/.github/workflows/release.yml@refs/heads/main$" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com -o text | head -25
  5. Pull the SBOM back out of the registry and confirm it describes the image you are holding. This is the workflow an auditor or a downstream consumer actually performs.
    bash
    cosign verify-attestation "ghcr.io/$REPO@$DIGEST" --type cyclonedx \
  --certificate-identity-regexp "^https://github.com/$REPO/.github/workflows/release.yml@refs/heads/main$" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  | jq -r '.payload' | base64 -d | jq '.predicate.components | length'
# And find one package by name:
cosign verify-attestation "ghcr.io/$REPO@$DIGEST" --type cyclonedx \
  --certificate-identity-regexp "^https://github.com/$REPO/.*" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  | jq -r '.payload' | base64 -d | jq -r '.predicate.components[] | select(.name=="fastapi") | .version'
    jq is doing the work here because an attestation is a signed DSSE envelope wrapping a base64 in-toto statement. Seeing the layers once makes the rest of the ecosystem legible.
Phase 3

Provenance: prove which build made it

GitHub issues SLSA v1 provenance for the image digest, tying it to the repository, the commit, the workflow file and the run — and you verify it with one command.

  1. A signature says "a workflow in this repo signed this". Provenance says "*this* commit, built by *this* workflow file, in *this* run, from *these* inputs". Add the attestation step to the build job, right after the push.
    yaml
          - name: Attest build provenance
        uses: actions/attest-build-provenance@v4
        with:
          subject-name: ghcr.io/${{ github.repository }}
          subject-digest: ${{ steps.push.outputs.digest }}
          push-to-registry: true
    push-to-registry: true stores the attestation next to the image in GHCR, so anyone who can pull the image can verify it. Without it the attestation lives only in the GitHub API, which is fine inside your organisation but useless to a downstream consumer.
  2. Merge, then verify the provenance with the GitHub CLI. Read the output: it names the workflow, the commit and the run.
    bash
    git switch -c feat/provenance && git add .github && git commit -m "release: SLSA build provenance"
git push -u origin feat/provenance && gh pr create --fill && gh pr merge --squash --delete-branch
git switch main && git pull && gh run watch
gh attestation verify "oci://ghcr.io/$REPO:latest" --repo "$REPO" --format json \
  | jq -r '.[0].verificationResult.statement.predicate.buildDefinition.externalParameters,
           .[0].verificationResult.statement.predicate.runDetails.builder.id'
  3. Now tighten it: verify not just that *some* workflow in the repo built it, but that the release workflow on main did. This is the assertion your deploy gate will make.
    bash
    gh attestation verify "oci://ghcr.io/$REPO:latest" \
  --repo "$REPO" \
  --signer-workflow "$REPO/.github/workflows/release.yml" \
  --source-ref refs/heads/main
echo "exit: $?"
# The same assertion, deliberately wrong, must fail:
gh attestation verify "oci://ghcr.io/$REPO:latest" --repo "$REPO" \
  --signer-workflow "$REPO/.github/workflows/nonexistent.yml"; echo "exit: $?"
  4. Record what the image now carries. A short table in security/SUPPLY-CHAIN.md that a colleague can follow is worth more than a diagram.
    text
    # security/SUPPLY-CHAIN.md

## What every image carries
| Artefact        | Produced by                          | Verify with                                  |
|-----------------|--------------------------------------|----------------------------------------------|
| Signature       | `cosign sign` (keyless, GH OIDC)     | `cosign verify --certificate-identity-regexp …` |
| SBOM (CycloneDX)| `syft` → `cosign attest --type cyclonedx` | `cosign verify-attestation --type cyclonedx` |
| SLSA provenance | `actions/attest-build-provenance@v4` | `gh attestation verify oci://… --repo … --signer-workflow …` |

## Identity we trust
issuer:   https://token.actions.githubusercontent.com
identity: https://github.com/<owner>/zero-to-prod/.github/workflows/release.yml@refs/heads/main

Anything signed by a different workflow, a different ref, or a different repository is not ours,
even if it is in our registry.
Phase 4

Harden the build itself

The pipeline that produces all this evidence cannot itself be the weak link: actions pinned by commit SHA, minimal token permissions, protected branches, and no path by which a fork's pull request can run privileged code.

  1. Pin every third-party action to a commit SHA. A tag is mutable: @v4 today and @v4 tomorrow can be different code, and that is precisely how several real compromises were delivered. Resolve each tag to its SHA and rewrite the workflows.
    bash
    git switch -c feat/pin-actions
pin() { sha=$(gh api "repos/$1/commits/$2" -q .sha); echo "$1@$sha # $2"; }
for a in "sigstore/cosign-installer v4.1.2" "anchore/sbom-action v0.24.2" \
         "aquasecurity/trivy-action 0.29.0" "google/osv-scanner-action v2.6.0" \
         "docker/build-push-action v6" "docker/login-action v3" "docker/metadata-action v5"; do
  pin $a
done
    actions/* and github/* are published by GitHub itself and are usually accepted at the tag; everything else gets a SHA with the tag in a trailing comment so Dependabot can still propose updates. Decide your own line and write it in SUPPLY-CHAIN.md.
  2. Edit the workflows to use the pinned references, then add a CI check that fails if an unpinned third-party action appears again.
    yaml
    # .github/workflows/supply-chain.yml — add this job
  pinned-actions:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Third-party actions must be pinned to a SHA
        run: |
          bad=$(grep -rhoE 'uses: [A-Za-z0-9_.-]+/[A-Za-z0-9_./-]+@[^ ]+' .github/workflows \
                | grep -vE '@[0-9a-f]{40}' \
                | grep -vE 'uses: (actions|github)/' || true)
          if [ -n "$bad" ]; then
            echo "unpinned third-party actions:"; echo "$bad"; exit 1
          fi
          echo "all third-party actions are pinned"
  3. Check the token permissions of every job. The default is often far more than a job needs; each workflow should declare a read-only default at the top and widen it per job.
    bash
    grep -n -A4 '^permissions:\|^  permissions:' .github/workflows/*.yml
# Rules of thumb:
#   default at the top of every file:  permissions: {contents: read}
#   build/push to GHCR:                packages: write
#   keyless signing / provenance:      id-token: write, attestations: write
#   posting a PR comment:              pull-requests: write  (only on that job)
  4. Close the fork-PR hole. pull_request_target runs with the base repository's secrets and a writable token — combined with checking out the PR's code it is a full repository compromise. Confirm you have none, and require approval for first-time contributors' workflow runs.
    bash
    grep -rn 'pull_request_target' .github/workflows/ && echo 'REVIEW THESE' || echo 'none — good'
gh api -X PUT "repos/$REPO/actions/permissions/workflow" \
  -f default_workflow_permissions=read -F can_approve_pull_request_reviews=false
gh api "repos/$REPO/actions/permissions/workflow"
# In the web UI: Settings → Actions → General → "Require approval for all external contributors"
  5. Protect the branch that the signing identity is bound to. The whole scheme rests on refs/heads/main meaning something: if anyone can push to main, anyone can produce a legitimately-signed backdoored image.
    bash
    gh api -X PUT "repos/$REPO/branches/main/protection" --input - <<'EOF'
{
  "required_status_checks": {"strict": true, "contexts": ["test", "pinned-actions"]},
  "enforce_admins": true,
  "required_pull_request_reviews": {"required_approving_review_count": 1},
  "restrictions": null,
  "allow_force_pushes": false,
  "allow_deletions": false
}
EOF
gh api "repos/$REPO/branches/main/protection" -q '{checks: .required_status_checks.contexts, force: .allow_force_pushes.enabled}'
    On a solo project the review requirement means you cannot merge your own PR, which is annoying — set required_approving_review_count to 0 and keep the status checks, or add a second account. Write down which you chose and why; that sentence is the security review.
  6. Merge the hardening and confirm the new check runs.
    bash
    git add .github && git commit -m "supply-chain: pin actions to SHAs, least-privilege tokens, pinned-actions check"
git push -u origin feat/pin-actions && gh pr create --fill && gh pr checks --watch
gh pr merge --squash --delete-branch && git switch main && git pull
Phase 5

The gate: refuse what you cannot verify

Deployment verifies the signature, the provenance and the identity before anything runs, and fails closed.

  1. Write the gate as a script, so it works identically on your laptop, on the EC2 host and in CI. It takes a full image reference, resolves it to a digest, and makes three assertions.
    bash
    # security/verify-image.sh
#!/usr/bin/env bash
set -euo pipefail

IMAGE="${1:?usage: verify-image.sh ghcr.io/owner/repo:tag}"
: "${EXPECTED_REPO:?set EXPECTED_REPO=owner/repo}"
WORKFLOW="${EXPECTED_WORKFLOW:-.github/workflows/release.yml}"
REF="${EXPECTED_REF:-refs/heads/main}"
ISSUER="https://token.actions.githubusercontent.com"
IDENTITY="^https://github.com/${EXPECTED_REPO}/${WORKFLOW}@${REF}$"

DIGEST=$(docker buildx imagetools inspect "$IMAGE" --format '{{.Manifest.Digest}}')
REF_BY_DIGEST="${IMAGE%%:*}@${DIGEST}"
echo "verifying ${REF_BY_DIGEST}"

echo "1/3 signature"
cosign verify "$REF_BY_DIGEST" \
  --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" > /dev/null

echo "2/3 SBOM attestation"
cosign verify-attestation "$REF_BY_DIGEST" --type cyclonedx \
  --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" > /dev/null

echo "3/3 build provenance"
gh attestation verify "oci://${REF_BY_DIGEST}" \
  --repo "$EXPECTED_REPO" --signer-workflow "${EXPECTED_REPO}/${WORKFLOW}" --source-ref "$REF" > /dev/null

echo "OK ${REF_BY_DIGEST}"
echo "$DIGEST"
  2. Make the deploy use it. Nothing pulls and runs without passing the gate first, and the gate's output — the digest — is what actually gets run, so a tag cannot be swapped between verification and use.
    bash
    # deploy/pull-and-run.sh
#!/usr/bin/env bash
set -euo pipefail
export EXPECTED_REPO="${EXPECTED_REPO:?}"
IMAGE="ghcr.io/${EXPECTED_REPO}:${1:-latest}"

DIGEST=$(../security/verify-image.sh "$IMAGE" | tail -1)
export APP_IMAGE="ghcr.io/${EXPECTED_REPO}@${DIGEST}"
echo "deploying $APP_IMAGE"
docker compose pull app
docker compose up -d app
docker compose ps app
  3. Test the gate against the real image — it should pass — and then against something it must refuse: a public image nobody signed with your identity.
    bash
    chmod +x security/verify-image.sh deploy/pull-and-run.sh
export EXPECTED_REPO=$(gh api user -q .login)/zero-to-prod
security/verify-image.sh "ghcr.io/$EXPECTED_REPO:latest"
security/verify-image.sh ghcr.io/sigstore/cosign/cosign:v3.1.3 || echo "refused, as it should be (exit $?)"
  4. Add the gate to the deploy job in CI too, before the SSM command that restarts the service on the EC2 host. The job that deploys is a different job from the one that builds, so it verifies rather than trusts.
    yaml
      deploy:
    needs: [build, sign]
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: read
      id-token: write
    env:
      GH_TOKEN: ${{ github.token }}
      EXPECTED_REPO: ${{ github.repository }}
    steps:
      - uses: actions/checkout@v4
      - uses: sigstore/cosign-installer@v4
        with: {cosign-release: v3.1.3}
      - name: Verify before deploying
        run: ./security/verify-image.sh "ghcr.io/${{ github.repository }}@${{ needs.build.outputs.digest }}"
      # ... existing aws-actions/configure-aws-credentials + SSM send-command steps ...
    verify-image.sh resolves a digest reference to itself, so passing image@sha256:… works unchanged.
Phase 6

Attack your own pipeline

Four attacks, four refusals, each traced to a named control — and one deliberate gap you write down instead of pretending it does not exist.

  1. Attack 1 — tamper with the image. Build a modified image, push it under a new tag in your own registry, and try to deploy it. It is *your* registry and *your* repository, and the gate still refuses, because nothing signed that digest.
    bash
    printf '\n# backdoor marker\nRUN echo "pwned" > /tmp/marker\n' >> Dockerfile.evil.tmp
cat Dockerfile Dockerfile.evil.tmp > Dockerfile.evil && rm Dockerfile.evil.tmp
docker build -f Dockerfile.evil -t "ghcr.io/$EXPECTED_REPO:evil" .
docker push "ghcr.io/$EXPECTED_REPO:evil"
security/verify-image.sh "ghcr.io/$EXPECTED_REPO:evil" || echo "REFUSED at step 1 (no signature for this digest)"
docker rm -f $(docker create "ghcr.io/$EXPECTED_REPO:evil") >/dev/null 2>&1 || true
rm Dockerfile.evil
  2. Attack 2 — move the tag. Retag the evil image as latest in the registry, the classic "the tag you deploy is not the image you reviewed" move. The gate resolves latest to its *current* digest, finds no signature for it, and refuses. Then restore the tag.
    bash
    GOOD=$(docker buildx imagetools inspect "ghcr.io/$EXPECTED_REPO:latest" --format '{{.Manifest.Digest}}')
echo "good digest: $GOOD"
docker buildx imagetools create -t "ghcr.io/$EXPECTED_REPO:latest" "ghcr.io/$EXPECTED_REPO:evil"
security/verify-image.sh "ghcr.io/$EXPECTED_REPO:latest" || echo "REFUSED (tag moved to an unsigned digest)"
docker buildx imagetools create -t "ghcr.io/$EXPECTED_REPO:latest" "ghcr.io/$EXPECTED_REPO@$GOOD"
security/verify-image.sh "ghcr.io/$EXPECTED_REPO:latest" && echo "restored"
    This is the attack that pinning by digest prevents entirely, which is why pull-and-run.sh deploys the digest the gate returned rather than the tag it was given.
  3. Attack 3 — build from a branch. Push a branch that changes the app and let the release workflow run on it (temporarily allow it, if your workflow is main-only). The image is signed — by an identity ending @refs/heads/attack — and the gate refuses it because the ref does not match.
    bash
    git switch -c attack/side-door
sed -i.bak 's/hello from the pipeline/hello from the side door/' app/main.py && rm app/main.py.bak
git commit -am "attack: build from a non-main branch" && git push -u origin attack/side-door
gh workflow run release.yml --ref attack/side-door 2>/dev/null || echo "workflow is main-only — that is control #1 working"
# If it did run, verify the resulting image:
security/verify-image.sh "ghcr.io/$EXPECTED_REPO:sha-$(git rev-parse --short HEAD)" || echo "REFUSED (identity ref is not refs/heads/main)"
git switch main && git push origin --delete attack/side-door && git branch -D attack/side-door
  4. Attack 4 — a dependency with a known vulnerability. Add an old, known-vulnerable package version and open a pull request; dependency review blocks it before a human even looks.
    bash
    git switch -c attack/bad-dep
# A version with published advisories; any old release of a scanned ecosystem works:
sed -i.bak 's/^dependencies = \[/dependencies = [\n  "requests==2.19.1",/' pyproject.toml && rm pyproject.toml.bak
uv pip compile pyproject.toml --generate-hashes -o requirements.lock
git commit -am "attack: pull in a known-vulnerable dependency" && git push -u origin attack/bad-dep
gh pr create --fill && gh pr checks --watch || echo "BLOCKED by dependency review / OSV"
gh pr close --delete-branch && git switch main
  5. Write up all four in security/SUPPLY-CHAIN.md, with the control that stopped each one and the evidence (a run URL, a command output). Then add the honest gap: the build runs on GitHub-hosted runners you do not control, and provenance proves *which workflow* built the image, not that the runner was uncompromised. Name what you would do about it (hardened runners, SLSA level 3 builders, reproducible builds) and why you are not doing it today.
    text
    ## Attacks run against this pipeline (2026-09-21)

| # | Attack                                   | Stopped by                                  | Evidence                    |
|---|------------------------------------------|---------------------------------------------|-----------------------------|
| 1 | Tampered image pushed to our registry     | `cosign verify` — no signature for digest    | verify-image.sh output       |
| 2 | `latest` retagged to the tampered image   | gate resolves the tag, then verifies digest  | verify-image.sh output       |
| 3 | Image built from a non-main branch        | identity regexp pins `@refs/heads/main`      | run #…, gate output          |
| 4 | Known-vulnerable dependency in a PR       | dependency review (fail-on-severity: high)   | PR #…                        |

## Known gaps
- **Runner trust.** Provenance attests which workflow ran, not that the runner was clean. Mitigation if the
  threat model warranted it: a hardened or self-hosted runner with egress policy, or a SLSA L3 builder.
- **Base image.** We pin `python:3.12-slim` by tag, not by digest; a rebuilt base changes our image silently.
  Fix: pin the base by digest and let Dependabot propose the bumps. (Next PR.)
- **Secrets in the build.** None today. If one is ever added, the SBOM and provenance do not detect leakage —
  that is what the secret scanning in the secure-service project covers.
  6. Close the base-image gap you just wrote down, because it takes two minutes and makes the point that a gap list is a to-do list.
    bash
    git switch -c fix/pin-base-image
BASE=$(docker buildx imagetools inspect python:3.12-slim --format '{{.Manifest.Digest}}')
sed -i.bak "s#FROM python:3.12-slim#FROM python:3.12-slim@$BASE#g" Dockerfile && rm Dockerfile.bak
grep '^FROM' Dockerfile
git commit -am "security: pin the base image by digest" && git push -u origin fix/pin-base-image
gh pr create --fill && gh pr merge --squash --delete-branch && git switch main && git pull
Phase 7

Optional: let the cluster enforce it

On the GitOps cluster, admission control refuses any image that is not signed by your workflow — so the gate is not something a deploy script can skip.

  1. Install Kyverno as another Argo CD application, the same way the GitOps project installs everything else.
    bash
    cd ../zero-to-prod-config && git switch -c feat/kyverno
mkdir -p platform/kyverno
cat > platform/kyverno/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kyverno
resources:
  - https://github.com/kyverno/kyverno/releases/download/v1.19.1/install.yaml
EOF
# plus argocd/applications/kyverno.yaml, modelled on the argo-rollouts one
  2. Write the policy. It requires a keyless Sigstore signature from your release workflow on every image in the orders-* namespaces, and mutates the image reference to its digest so the admitted pod runs exactly what was verified.
    yaml
    # platform/kyverno/verify-images.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-orders-api-images
spec:
  validationFailureAction: Enforce
  background: false
  webhookTimeoutSeconds: 30
  rules:
    - name: signed-by-our-release-workflow
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [orders-dev, orders-prod]
      verifyImages:
        - imageReferences:
            - "ghcr.io/YOUR_GITHUB_USER/zero-to-prod*"
          mutateDigest: true
          required: true
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/YOUR_GITHUB_USER/zero-to-prod/.github/workflows/release.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev
  3. Merge, wait for Argo CD to sync, then prove it: an unsigned image is rejected at admission with the policy's message, and the signed one is admitted with its tag rewritten to a digest.
    bash
    git add platform/kyverno argocd/applications/kyverno.yaml
git commit -m "platform: kyverno image verification for orders namespaces"
git push -u origin feat/kyverno && gh pr create --fill && gh pr merge --squash --delete-branch
git switch main && git pull && argocd app wait kyverno --sync --health --timeout 300
kubectl -n orders-dev run unsigned --image=nginx:alpine --restart=Never || echo "REFUSED by admission control"
kubectl -n orders-dev get pod -l app.kubernetes.io/instance=orders-api -o jsonpath='{.items[0].spec.containers[0].image}{"\n"}'
    The last command should print an image ending in @sha256:… — Kyverno's mutateDigest rewrote the tag at admission. Now "what is running" and "what was verified" are the same string.
Help

Troubleshooting

cosign sign fails in CI with getting key from Fulcio: no identity token
The job is missing id-token: write in its permissions block, or the permissions are set at workflow level but overridden at job level. Both the signing job and any job running actions/attest-build-provenance need it.
cosign verify says no matching signatures for an image you know was signed
Almost always the identity regexp. Print what is actually there: cosign verify <image> --certificate-identity-regexp '.*' --certificate-oidc-issuer https://token.actions.githubusercontent.com -o text and read the Subject line, then build your regexp from it. A . in a workflow path is a regex wildcard — anchor with ^…$ and remember the ref suffix.
gh attestation verify reports no attestations found
Either the attestation was not pushed to the registry (push-to-registry: true) and you are verifying oci://, or you are verifying a tag whose digest has changed since the build. Verify by digest. Note also that attestations are per-digest: a multi-architecture image's index digest and its per-platform digests are different subjects.
Trivy fails the build on a critical you cannot fix
Check that ignore-unfixed: true is set. If the CVE does have a fix, the fix is to rebuild on a newer base image — docker pull python:3.12-slim and re-pin the digest. If you must ship anyway, add a dated, justified entry to .trivyignore with an expiry, and review it; a permanent ignore is a permanent hole.
pip install --require-hashes fails with "In --require-hashes mode, all requirements must have their versions pinned"
A transitive dependency is missing from the lock file, usually because uv pip compile was run against a different Python version or platform than the image uses. Compile inside the same base image: docker run --rm -v "$PWD:/w" -w /w python:3.12-slim sh -c 'pip install uv && uv pip compile pyproject.toml --generate-hashes -o requirements.lock'.
Kyverno rejects every image, including the signed one, with a timeout
The cluster cannot reach Rekor or Fulcio; in a kind cluster behind a proxy this is common. Check the Kyverno pod logs, and confirm egress works: kubectl -n kyverno exec deploy/kyverno-admission-controller -- wget -qO- https://rekor.sigstore.dev/api/v1/log | head -c 100.
Next

Where to go from here

Did a step fail or feel unclear? Tell me which one →