Education › Security › Stage 2: Securing systems

Container & Kubernetes security

Image provenance, non-root containers, RBAC, network policies, pod security standards, admission control.

Intermediate ~40 min read Module 8 of 16

Containers changed where the security boundaries are. The image is now the operating system, the pod is the process boundary, and the cluster API is the most powerful thing on the network. A container is not a virtual machine: it is a process with a different view of the filesystem and namespaces, sharing the host kernel, and a container running as root with a mounted socket is root on the host. This module covers the four layers that matter — the image you build, the runtime settings of the pod, the cluster's access and network policies, and the admission controls that make the good configuration the only configuration — with the settings that close the paths attackers actually use.

After this module you can
  • Build minimal, non-root, pinned images and scan them for known vulnerabilities before they run
  • Apply Pod Security Standards: no privilege escalation, dropped capabilities, read-only root filesystem, no host namespaces
  • Design RBAC and service accounts so a compromised pod cannot become a compromised cluster
  • Write default-deny NetworkPolicies and open only the flows the workload needs
  • Enforce these controls with admission policies and verify image provenance

The image is the attack surface

Every package in an image is code an attacker can use once inside. A full distribution base image ships hundreds of packages, a shell, a package manager and a compiler; a distroless or slim base ships your runtime and little else. Smaller images have fewer vulnerabilities to patch, fewer tools for an attacker to live off, and faster pulls. Pin the base by digest, not by a floating tag, so that a build today and a build next month contain the same bytes until you choose to update.

A hardened multi-stage image: build tools stay in the builder, the final stage is distroless, non-root, and pinned by digest.
dockerfile
# syntax=docker/dockerfile:1
FROM python:3.12-slim@sha256:0d0a3a2b4d1c7b5f6e8c9a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d AS build
WORKDIR /src
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
COPY app ./app

FROM gcr.io/distroless/python3-debian12:nonroot
COPY --from=build /install /usr/local
COPY --from=build /src/app /app/app
WORKDIR /app
USER nonroot:nonroot
EXPOSE 8000
ENTRYPOINT ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Scan every image in CI and block on critical findings with a fix available; scan again on a schedule, because vulnerabilities are published after you built. Rebuild images regularly rather than patching containers in place, and never bake secrets into layers (the secrets module). Sign images at build time and verify the signature before they run, so that only images your pipeline produced can be scheduled — the supply chain module covers the signing tooling.

Scanning an image with Trivy in CI: fail on critical and high vulnerabilities that have a fix.
bash
trivy image --severity CRITICAL,HIGH --ignore-unfixed --exit-code 1 ghcr.io/acme/orders:sha-1a2b3c4

# what is actually in the image, for the SBOM and for curiosity
trivy image --format table --list-all-pkgs ghcr.io/acme/orders:sha-1a2b3c4 | head -40
Tip

Digest pins look ugly and are worth it. Use a bot (Renovate or Dependabot) to open pull requests when the base image publishes a new digest, so pinning does not mean falling behind.

Runtime: what the pod may do

A pod's securityContext decides how far a compromise inside the container can reach. The defaults are permissive: root user, all default capabilities, a writable root filesystem, and the ability to gain privileges via setuid binaries. The Pod Security Standards define three levels (privileged, baseline, restricted); the restricted profile is what production workloads should meet, and the built-in Pod Security Admission controller can enforce it per namespace with a label.

A pod spec that satisfies the restricted profile. Most applications run unchanged with these settings; those that write to disk get an emptyDir for the path.
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
spec:
  template:
    spec:
      serviceAccountName: orders
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: ghcr.io/acme/orders@sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests: {cpu: "100m", memory: "128Mi"}
            limits: {memory: "256Mi"}
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

Never mount the host's Docker or containerd socket, never use hostPID, hostNetwork or hostPath for application workloads, and never set privileged: true outside a handful of infrastructure components you can name. Each of those is a container escape by design. Resource limits are security too: a pod without a memory limit can take the node down, which is denial of service by accident or on purpose.

Enforce the restricted profile for a namespace with Pod Security Admission; warn first, then enforce.
bash
kubectl label namespace shop \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted
# once the warnings are gone:
kubectl label namespace shop pod-security.kubernetes.io/enforce=restricted --overwrite

Identity and RBAC: the cluster API is the crown jewel

Every pod runs with a service account, and by default its token is mounted into the container. A compromised pod with a token that can list secrets or create pods owns the cluster. Give each workload its own service account, do not mount the token unless the application talks to the API, and bind roles that grant exactly the verbs on exactly the resources it needs, in its own namespace. Cluster-wide roles are for controllers, not applications.

A role that lets the orders service read one ConfigMap and nothing else, bound to its own service account.
yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: shop
  name: orders-config-reader
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["orders-config"]
    verbs: ["get", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: shop
  name: orders-config-reader
subjects:
  - kind: ServiceAccount
    name: orders
    namespace: shop
roleRef:
  kind: Role
  name: orders-config-reader
  apiGroup: rbac.authorization.k8s.io

For humans, federate cluster access to your identity provider (OIDC) with groups mapped to roles, so that leaving the company removes cluster access automatically, and audit the API server: who created, deleted or exec'd into what. kubectl exec into production pods should be rare, logged and, ideally, gated by an approval. The commands below are how you find privilege you did not intend to grant.

Auditing RBAC: who can do the dangerous things.
bash
# can this service account read secrets cluster-wide? (should be no)
kubectl auth can-i list secrets --all-namespaces --as=system:serviceaccount:shop:orders

# every binding to cluster-admin
kubectl get clusterrolebindings -o json \
  | jq -r '.items[] | select(.roleRef.name=="cluster-admin") | .metadata.name + ": " + (.subjects // [] | map(.kind + "/" + .name) | join(", "))'

# roles that allow creating pods or exec (paths to escalation)
kubectl get roles,clusterroles -A -o json \
  | jq -r '.items[] | select(.rules[]? | (.resources // []) | index("pods") or index("pods/exec")) | .metadata.namespace + "/" + .metadata.name' | sort -u

Network policy: zones inside the cluster

By default every pod can reach every other pod and the internet. A NetworkPolicy is the cluster equivalent of a security group: it selects pods by label and lists allowed ingress and egress. Start each namespace with a default-deny policy for both directions, then add allows per workload: the API accepts traffic from the ingress controller, talks to its database and its cache, resolves DNS, and reaches the external payment API through the egress gateway — nothing else. Your CNI must support policies for them to have any effect; check before relying on them.

Default deny for a namespace, then the orders pod's explicit allows. DNS egress is the one everybody forgets.
yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: orders
  namespace: shop
spec:
  podSelector:
    matchLabels: {app: orders}
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: {kubernetes.io/metadata.name: ingress-nginx}
      ports: [{protocol: TCP, port: 8000}]
  egress:
    - to:
        - podSelector:
            matchLabels: {app: postgres}
      ports: [{protocol: TCP, port: 5432}]
    - to:
        - namespaceSelector:
            matchLabels: {kubernetes.io/metadata.name: kube-system}
          podSelector:
            matchLabels: {k8s-app: kube-dns}
      ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
Note

A service mesh adds mTLS and identity-based authorisation on top of network policy: policies say which workload may call which endpoint, not just which IP may reach which port. Network policy is the floor; the mesh is the next layer.

Admission: make the right configuration the only one

Documentation asks; admission control enforces. Every object sent to the API passes through admission before it is stored, and a policy engine there can reject what violates the rules: images not from your registry, images without a valid signature, containers without resource limits, pods that run as root, latest tags, missing labels. Kubernetes has built-in Validating Admission Policies (CEL expressions), and engines such as Kyverno or OPA Gatekeeper add richer policy libraries, mutation and reporting.

A Kyverno policy: only signed images from the company registry may run. Audit first, then enforce.
yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: only-signed-company-images
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [shop, payments]
      verifyImages:
        - imageReferences: ["ghcr.io/acme/*"]
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/acme/*/.github/workflows/release.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
          mutateDigest: true
          required: true
POD: RESTRICTED PROFILE, NO HOST ACCESSescape blockedlist secrets? deniedgates every objectdeniedallowed egressAttackercode runs in the podDistroless imageno shell, no toolssecurityContextnon-root, no capsService accountscoped RoleCluster APIRBAC decidesAdmission policysigned images, limitsNodekernel, other podsOnly: db, DNSNetworkPolicyOther namespacesunreachable
Five independent layers between an attacker who gets code running in a pod and the rest of the cluster: the image limits what tools exist, the pod's security context limits what the process can do, RBAC limits what the API will allow, network policy limits who it can reach, and admission control makes sure every workload has the other four.

Pair admission with runtime detection: a tool such as Falco watches system calls and flags a shell spawned in a container, a process reading /etc/shadow, or an outbound connection from a pod that never makes them. Prevention stops the known bad; detection tells you about the rest. Together with image scanning, restricted pods, scoped RBAC and network policy, you have five independent layers, and an attacker needs to defeat all of them.

Hands-on practice

Harden a workload on a local cluster

  1. Create a kind cluster with a CNI that supports NetworkPolicy (install Calico or Cilium after kind create cluster --config with the default CNI disabled). Create namespace shop.
  2. Rebuild the zero-to-production API image using the distroless, non-root Dockerfile from the lesson. Run trivy image against the old and new image and compare vulnerability counts.
  3. Deploy the hardened Deployment spec. Label the namespace with enforce=restricted and confirm that a pod with privileged: true is rejected.
  4. Create the orders service account, Role and RoleBinding. Run kubectl auth can-i --list --as=system:serviceaccount:shop:orders and confirm it can only get the ConfigMap.
  5. Apply the default-deny and orders NetworkPolicies. From a debug pod in the namespace, confirm you cannot reach the orders pod, and from the orders pod confirm DNS still resolves and Postgres is reachable.
  6. Install Kyverno and apply a policy that rejects latest tags and containers without memory limits in audit mode; read the policy report, then switch to enforce.
  7. Delete the cluster.
Cheat sheet

Container & Kubernetes security — at a glance

Main things to focus on

  • Minimal, non-root, digest-pinned images; scan in CI and on a schedule; rebuild rather than patch
  • Restricted Pod Security Standard: runAsNonRoot, no privilege escalation, drop ALL capabilities, read-only root, seccomp
  • No host namespaces, no privileged, no Docker socket, memory limits on everything
  • One service account per workload, token not mounted unless needed, Role scoped to named resources
  • Default-deny NetworkPolicy per namespace, explicit allows including DNS
  • Admission enforces: signed images from your registry, no latest, limits present; runtime detection watches the rest

Image

FROM image@sha256:...Pin base by digest; update via bot PRs
gcr.io/distroless/*:nonrootNo shell, no package manager, non-root user
USER nonroot / USER 65532Never run as root in the final stage
trivy image --severity CRITICAL,HIGH --ignore-unfixed --exit-code 1 IMGBlock on fixable serious findings
multi-stage buildCompilers and build tools never reach production

Pod securityContext

runAsNonRoot: true / runAsUser: 65532Refuse to start as root
allowPrivilegeEscalation: falseNo setuid gains
capabilities: {drop: [ALL]}Add back NET_BIND_SERVICE only if truly needed
readOnlyRootFilesystem: true + emptyDir /tmpNothing persists or gets dropped into the image
seccompProfile: {type: RuntimeDefault}Block dangerous syscalls
automountServiceAccountToken: falseNo API token unless the app uses the API
pod-security.kubernetes.io/enforce=restrictedNamespace label enforcing the profile

RBAC

Role + RoleBinding in namespace, resourceNames: [...]Least privilege per workload
kubectl auth can-i VERB RESOURCE --as=system:serviceaccount:NS:SATest what an identity can do
kubectl auth can-i --list --as=...Everything an identity can do
clusterrolebindings -> cluster-adminAudit who is cluster admin; should be tiny
OIDC groups -> rolesHuman access tied to the identity provider

NetworkPolicy

podSelector: {} + policyTypes [Ingress, Egress]Default deny for the namespace
namespaceSelector: kubernetes.io/metadata.name: NSAllow from a specific namespace
egress to kube-dns on 53 UDP/TCPDo not forget DNS
ipBlock: {cidr: ...}External destinations by CIDR (prefer an egress gateway)
check CNI supports policyOtherwise policies are silently ignored

Admission and runtime

Kyverno verifyImages keyless (issuer, subject)Only images signed by your pipeline
validationFailureAction: Audit -> EnforceReport first, then block
policies: no :latest, limits required, registry allow-listThe usual starter set
Falco rules: shell in container, sensitive file readRuntime detection of what got through

Common pitfalls

  • Running as root in the container because the base image does, then mounting a hostPath.
  • Mounting the Docker or containerd socket into a CI runner pod; that is root on the node.
  • A shared service account with a broad ClusterRole because one job needed it once.
  • Applying default-deny without the DNS egress rule and concluding that network policy is broken.
  • Pinning images by tag and being surprised when python:3.12-slim changed under you.
  • Writing security policy in a wiki instead of an admission controller.
Quiz

Check your understanding

5 questions · 4 to pass · answers are explained as you go. Your best score is saved on this device only.

Progress and quiz scores are saved in this browser only. Back up or restore on the hub.

Was this lesson useful? Tell me what to improve →