Most real-world compromises of delivery systems do not involve clever exploits. They involve a credential that was somewhere it should not have been: an access key in a Git history, a token in a CI log, a password shared in chat. Pipelines and clusters concentrate powerful credentials, which makes them targets. This module covers how to take secrets out of code, replace long-lived keys with short-lived identity, and build security checks into the pipeline so that they run on every change instead of once a year.
- Identify where secrets leak, and respond correctly when one does
- Store and deliver secrets with a secret manager instead of code, images or plain environment files
- Replace long-lived cloud keys in CI with OIDC federation and short-lived credentials
- Apply least privilege across IAM, Kubernetes RBAC and pipeline permissions
- Add automated security gates to a pipeline: secret scanning, SAST, dependency, IaC and policy-as-code checks
Where secrets leak, and what to do when they do
A secret is anything that grants access: passwords, API tokens, private keys, database URLs with credentials, signing keys, webhook secrets. They escape through a predictable set of routes.
- Source control. A
.envfile or a hard-coded key is committed. Deleting it in a later commit does not help, because it remains in the history and in every clone and fork. - Container images.
ENV,ARGandCOPYpersist in image layers, as the Docker module showed. - Logs and error messages. A debug line that prints the whole configuration, or a stack trace containing a connection string.
- CI configuration. Secrets echoed by a script, or exposed to workflows triggered by untrusted pull requests.
- Terraform state and Kubernetes Secrets, which hold values in plain text or in base64, which is encoding and not encryption.
- People. Credentials pasted into chat, tickets, wikis and screenshots.
When a secret leaks, the order of operations matters, and most teams get it wrong under pressure.
- Revoke or rotate the credential immediately. Assume it has been copied. Public repositories are scraped for keys within minutes.
- Check the audit logs for use of that credential from the moment of exposure.
- Then clean up the history, if it is worth doing at all. A rotated secret in the history is harmless; an unrotated one that was deleted is still live.
- Fix the cause: add scanning, move the secret to a manager, and write it up.
gitleaks detect --source . --verbose # scan the working tree and full Git history
gitleaks protect --staged # scan only what is about to be committed
# run it on every commit with the pre-commit framework
pre-commit installTurn on your Git host's built-in secret scanning and push protection, which rejects a push that contains a recognisable credential. Blocking the secret before it lands is far cheaper than responding afterwards.
Secret managers
A secret manager is a service built for this one job: AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or HashiCorp Vault if you run your own. It provides what files and environment files cannot.
| Capability | Why it matters |
|---|---|
| Encryption at rest, with keys held in a KMS | A stolen disk or backup reveals nothing |
| Access controlled by IAM, per secret | The orders service can read its own database password and nothing else |
| An audit log of every read | After an incident you know exactly which secrets were accessed, and by whom |
| Versioning and rotation | Change a credential in one place, on a schedule, without redeploying every consumer |
| Dynamic secrets (Vault, some cloud databases) | Credentials created on demand for one client and expired automatically |
aws secretsmanager create-secret --name shop/prod/orders/db \
--secret-string '{"username":"orders","password":"REPLACE_ME"}'
aws secretsmanager get-secret-value --secret-id shop/prod/orders/db \
--query SecretString --output text | jq -r .passwordIn Kubernetes, the External Secrets Operator bridges the gap. You commit a manifest that names the entry in the secret manager; the operator, which has a role allowing it to read that entry, creates and refreshes an ordinary Kubernetes Secret. Git holds only a reference, which fits the GitOps module exactly.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: orders-db
namespace: shop
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets
kind: ClusterSecretStore
target:
name: orders-secrets # the Kubernetes Secret to create
data:
- secretKey: DATABASE_PASSWORD
remoteRef:
key: shop/prod/orders/db
property: passwordPrefer mounting secrets as files over environment variables where your application allows it. Environment variables are inherited by child processes and tend to appear in crash dumps and debug endpoints, and a mounted file can be refreshed without restarting the pod.
The operator's API version changes as the project matures. Check the version installed in your cluster with kubectl api-resources | grep externalsecret before copying a manifest.
OIDC: deploying without stored keys
The traditional way to let CI deploy to the cloud is to create a user, generate an access key, and paste it into the CI system's secrets. That key is long-lived, powerful, held by a third party, and rarely rotated. OIDC federation removes it.
- At the start of a job, the CI platform issues a signed, short-lived identity token. Its claims describe the job: which repository, which branch, which environment.
- The job presents the token to the cloud provider and asks to assume a role.
- The cloud verifies the signature against the CI platform's public keys, and checks the claims against the role's trust policy.
- If they match, it returns temporary credentials that expire, typically within an hour.
jobs:
deploy:
runs-on: ubuntu-latest
environment: production
permissions:
id-token: write # allow this job to request an OIDC token
contents: read
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/shop-deploy
aws-region: eu-west-1
- run: aws sts get-caller-identity{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:acme/shop:ref:refs/heads/main"
}
}
}
]
}The sub condition is the security boundary. A trust policy that checks only the audience, or uses a wildcard such as repo:acme/*, lets any repository or any branch in your organisation, including an unreviewed pull request branch, assume the role.
The same pattern gives Kubernetes pods cloud access without keys: workload identity on GKE and AKS, IAM roles for service accounts or Pod Identity on EKS. Each pod's service account is mapped to a cloud role, so the orders pod and the payments pod get different permissions.
Least privilege everywhere
Least privilege means every identity can do exactly what its job requires and nothing more, so that a compromise of one component is contained there. It applies at every layer you have met in this track.
- Cloud IAM. One role per workload, with specific actions on specific resources. Start from zero and add what fails, instead of starting from administrator and meaning to trim later.
- Pipelines. A
permissions:block on every workflow, and separate roles for plan and apply, and for staging and production. - Kubernetes RBAC. Humans get read access by default, and changes go through Git. Each workload gets its own ServiceAccount.
- Network. Security groups and Kubernetes NetworkPolicies that allow only the flows a service needs.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: shop
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: shop-developers-read
namespace: shop
subjects:
- kind: Group
name: shop-developers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.iokubectl auth can-i delete pods -n shop --as jane@example.com
kubectl auth can-i --list -n shop --as system:serviceaccount:shop:ordersBe careful with permissions that look harmless. In Kubernetes, the right to read Secrets in a namespace is the right to every credential there, and the right to create pods is, in effect, the right to use any ServiceAccount in that namespace.
DevSecOps: security gates in the pipeline
DevSecOps means security checks are automated, run on every change, and give feedback in the pull request, while the author still has the context to fix the problem. This is often called shifting left. It does not replace security review or penetration testing; it removes the routine findings so that experts can spend their time on the hard ones.
| Check | Finds | Example tools |
|---|---|---|
| Secret scanning | Credentials in code and history | gitleaks, host push protection |
| SAST (static analysis) | Injection, unsafe deserialisation, weak cryptography in your code | Semgrep, CodeQL |
| SCA (dependency scanning) | Known CVEs in libraries | Trivy, Grype, Dependabot alerts |
| Image scanning | CVEs in OS packages, running as root | Trivy, Grype |
| IaC scanning | Public buckets, open security groups, unencrypted disks | Checkov, Trivy config, tfsec |
| DAST (dynamic analysis) | Flaws visible in a running application | OWASP ZAP |
Policy as code takes this further: organisational rules become versioned, testable code that is enforced automatically. Open Policy Agent with its Rego language, and Kyverno with YAML policies, are the common tools. The same policy can run in CI against manifests, where it advises, and as a Kubernetes admission controller, where it blocks.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-run-as-non-root
spec:
validationFailureAction: Enforce
rules:
- name: check-run-as-non-root
match:
any:
- resources:
kinds: [Pod]
validate:
message: "Containers must set securityContext.runAsNonRoot to true."
pattern:
spec:
containers:
- securityContext:
runAsNonRoot: trueIntroduce gates gradually. Start every new check in report-only or Audit mode, fix the existing findings, and only then switch it to blocking for new violations. A gate that fails every build on day one will be disabled on day two, and a disabled check protects nothing.