A secret is any value whose disclosure lets someone act as you: an API key, a database password, a signing key, a session token. Secrets leak in boring ways — a commit, a log line, a screenshot, an environment dump in an error page — and the leak is usually discovered by an attacker first. This module is about the whole lifecycle: keeping secrets out of the places they leak from, storing and delivering them through a secrets manager, replacing long-lived secrets with short-lived ones wherever possible, rotating without downtime, and responding correctly when one gets out. The DevOps track introduced the tools; this module makes the practice systematic.
- Identify every place a secret can leak and close the common ones (git, logs, images, tickets, CI output)
- Deliver secrets to workloads from a secrets manager or KMS at runtime rather than baking them in
- Replace static credentials with short-lived, identity-derived ones (cloud roles, OIDC, dynamic database credentials)
- Rotate a secret without an outage using dual-validity
- Run the response to a leaked secret in the right order
Where secrets leak
Secrets do not leak from vaults. They leak from the edges: a .env file added to git, a debug log that prints the request headers, a Docker layer that copied a key and "deleted" it in the next layer, a CI job that echoed its environment, a Slack message, a stack trace on an error page, a public object storage bucket with a config backup. Public repositories are scraped continuously; a cloud key pushed to GitHub is typically used within minutes.
| Leak path | Control |
|---|---|
| Source code and git history | Pre-commit and CI secret scanning; .gitignore for env files; history rewrite is not enough, rotate |
| Container images | Never COPY secrets; multi-stage builds; use build secrets that do not persist in layers |
| Logs and error pages | Redact known secret shapes at the logger; generic error pages; never log headers or env |
| CI/CD output | Masked secrets, no env dumps, forked PRs get no secrets |
| Chat, tickets, wikis | A policy plus a scanner; share via the secrets manager's link, not the value |
# one-off scan of the whole history
gitleaks git --redact -v .
# pre-commit: block commits that contain a secret
cat > .pre-commit-config.yaml <<'EOF'
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.21.2
hooks:
- id: gitleaks
EOF
pre-commit installRemoving a secret from a later commit changes nothing: the value is in the history and in every clone. A leaked secret is a compromised secret; the only fix is revocation and rotation.
A secrets manager, and how workloads get secrets
A secrets manager (a cloud provider's secrets service, HashiCorp Vault, or similar) stores secrets encrypted with a KMS-backed key, controls access with identity-based policy, logs every read, versions values so rotation can roll back, and can generate secrets on demand. The application authenticates to it with its workload identity (an instance role, a Kubernetes service account, an OIDC token) and fetches what it needs at startup or on demand. Nothing is stored in code, images or CI variables except the identity that is allowed to ask.
import json
import boto3
_client = boto3.client("secretsmanager")
_cache: dict[str, dict] = {}
def get_secret(name: str) -> dict:
"""Return the secret's JSON fields; the instance/pod role must allow secretsmanager:GetSecretValue on it."""
if name not in _cache:
resp = _client.get_secret_value(SecretId=name)
_cache[name] = json.loads(resp["SecretString"])
return _cache[name]
db = get_secret("prod/orders/db")
# db["username"], db["password"], db["host"] -> build the connection; do not print themOn Kubernetes the same pattern is a CSI secrets-store driver or an operator (External Secrets) that syncs from the manager into Secret objects, with the manager remaining the source of truth. Plain Kubernetes Secrets are base64, not encrypted, unless encryption at rest is configured for etcd; treat them as delivery, not storage. Mount secrets as files with restrictive permissions rather than environment variables where you can: environment variables are inherited by child processes and appear in crash dumps and debug endpoints.
Scope access by path and identity: the orders service can read prod/orders/* and nothing else. A single "app" role with read on everything turns one compromised pod into all your secrets.
Short-lived beats well-guarded
The best secret is one that expires before it can be misused. Cloud instance and pod roles hand out credentials that rotate automatically every few hours. OIDC federation lets CI jobs exchange a signed job token for cloud credentials that last minutes. Dynamic database credentials go further: the secrets manager creates a database user with a lease when the application asks, and drops it when the lease expires, so there is no shared password to leak at all. Signed URLs and tokens with exp claims do the same for object access and APIs.
# configure once: how Vault connects, and a role that defines the SQL to create a user
vault write database/config/orders \
plugin_name=postgresql-database-plugin \
allowed_roles="orders-app" \
connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/orders" \
username="vault" password="REPLACED_AFTER_ROTATE_ROOT"
vault write database/roles/orders-app \
db_name=orders \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1h" max_ttl="24h"
# the application (authenticated by its identity) asks for credentials
vault read database/creds/orders-app
# username v-k8s-orders-app-3Hx... password A1b... lease_duration 1hWhere a static secret is unavoidable (a third-party API key, a legacy system password), shorten its life administratively: rotate it on a schedule, restrict where it can be used from (IP allow-lists, scoped permissions), and monitor its use so that use from a new place is an alert.
Rotation without an outage
Rotation fails in production when it is treated as a single swap: the new value is written, every consumer that still holds the old one breaks. The safe pattern is dual validity: create the new credential while the old one still works, roll consumers over to the new one, verify nothing uses the old one, then revoke it. For signing keys the same idea is key identifiers: publish both keys, sign with the new, verify with either, retire the old after all tokens signed by it have expired.
- Create the new secret (new key, new database user, new API key) alongside the old.
- Store it as the new version in the secrets manager; consumers that fetch on start pick it up on their next restart, consumers with a refresh loop pick it up within minutes.
- Roll consumers deliberately (rolling restart) and watch error rates and auth failures.
- Check the audit log: no successful use of the old version for a full cycle.
- Revoke the old secret. Keep the old version in the manager, disabled, for forensics.
Build consumers to handle rotation from the start: read secrets at startup, refresh on a timer or on an auth failure, and never cache a secret to disk. A service that needs a redeploy to pick up a new password will be the reason rotation is skipped.
When a secret leaks: the order of operations
Speed and order matter more than blame. Assume the secret has already been used. Revoke or rotate first, before investigating, because every minute it remains valid is a minute of attacker access. Then audit: search cloud and application logs for use of the credential since the earliest possible exposure, looking especially for use from unfamiliar sources. Contain what the audit shows: new keys created, resources touched, data read. Fix the leak path so it cannot recur (add the scanner, redact the logger). Finally write it up blamelessly; the person who leaked it and reported it fast did the right thing, and the process that let it happen is what needs to change.
# events by this access key in the last 24 hours (CloudTrail must be enabled)
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAEXAMPLEKEY \
--start-time "$(date -u -v-24H +%Y-%m-%dT%H:%M:%SZ)" \
--query 'Events[].{time:EventTime,name:EventName,src:CloudTrailEvent}' --output json \
| jq -r '.[] | "\(.time) \(.name) \(.src | fromjson | .sourceIPAddress)"' | sort | uniq -c | sort -rn | headPrepare the playbook before you need it: who can revoke which secret, where the audit logs are, which consumers must be restarted, and who is told. A leak handled in twenty minutes with a clear playbook is an incident; the same leak discovered a month later from a cloud bill is a breach.