Packaging with Helm

Charts, values and releases

A chart is a directory of templated Kubernetes manifests plus metadata. Values are the parameters you feed into those templates. A release is one installed instance of a chart in a cluster, with a name. Install the same chart twice with different names and values and you get two independent releases. Every upgrade or rollback of a release creates a new numbered revision, which Helm stores in the cluster as a Secret in the release's namespace.

orders/
  Chart.yaml            # name, chart version, appVersion, dependencies
  values.yaml           # default values: the chart's public interface
  templates/
    deployment.yaml
    service.yaml
    ingress.yaml
    _helpers.tpl        # named template snippets; files starting with _ render nothing
    NOTES.txt           # message printed after install
  charts/               # downloaded dependency charts

apiVersion: v2
name: orders
description: Orders API
type: application
version: 0.3.1          # version of the CHART (packaging); bump on any chart change
appVersion: "1.4.2"     # version of the APPLICATION it deploys; informational

Those two versions are independent, and mixing them up is a common confusion. Change a template and you bump version. Ship a new build of the app and appVersion, or more commonly the image tag in values, changes.

Using charts other people wrote

Much of your first Helm use is installing third-party software: an ingress controller, cert-manager, Prometheus. Charts are distributed through repositories or, increasingly, as OCI artifacts in the same registries that hold your images.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm search repo ingress-nginx --versions | head

helm show values ingress-nginx/ingress-nginx > defaults.yaml    # every knob the chart offers

helm upgrade --install ingress ingress-nginx/ingress-nginx \
  --namespace ingress --create-namespace \
  --version 4.11.3 \
  -f ingress-values.yaml

Three habits are worth forming at once. Use helm upgrade --install, which is idempotent: it installs if the release is missing and upgrades if it exists, so a pipeline can run the same command every time. Always pin --version, or you get whatever chart is newest on the day. And keep your overrides in a values file under version control instead of a growing list of --set flags.

Writing templates

Templates are Kubernetes YAML with Go template actions inside {{ }}. At render time Helm gives each template a set of built-in objects: .Values (merged values), .Release (name, namespace, revision), .Chart (the contents of Chart.yaml) and .Capabilities (what the cluster supports).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "orders.fullname" . }}
  labels:
    {{- include "orders.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "orders.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "orders.selectorLabels" . | nindent 8 }}
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
    spec:
      containers:
        - name: orders
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          ports:
            - containerPort: {{ .Values.service.targetPort }}
          {{- with .Values.resources }}
          resources:
            {{- toYaml . | nindent 12 }}
          {{- end }}

• Pipelines pass a value through functions, left to right: {{ .Values.image.tag | default .Chart.AppVersion | quote }}.
• Whitespace control: {{- trims whitespace before the action and -}} trims after it. YAML is indentation-sensitive, so this matters.
• toYaml plus nindent is how you drop a whole block from values (resources, tolerations, annotations) into the right indentation.
• include renders a named template from _helpers.tpl and, unlike the built-in template action, its output can be piped onward.
• with changes the scope and skips the block when the value is empty; if and range give conditionals and loops.
• The checksum annotation solves the problem from the Kubernetes module: when the ConfigMap's content changes, the pod template changes too, so pods are rolled automatically.

{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "orders.fullname" . }}
spec:
  ingressClassName: {{ .Values.ingress.className }}
  rules:
    {{- range .Values.ingress.hosts }}
    - host: {{ . | quote }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ include "orders.fullname" $ }}
                port:
                  number: {{ $.Values.service.port }}
    {{- end }}
{{- end }}

Inside range, the dot becomes the current item. Use $ to reach the root scope, as the example does for $.Values. Forgetting that is the most common template error.

One chart, many environments

values.yaml holds sensible defaults and documents the chart's interface. Each environment gets a small file containing only what differs. Values are merged in order, and later sources win: chart defaults, then each -f file from left to right, then --set flags.

replicaCount: 2
image:
  repository: ghcr.io/acme/orders-api
  tag: ""                 # empty means: use the chart's appVersion
service:
  port: 80
  targetPort: 8000
ingress:
  enabled: false
  className: nginx
  hosts: []
resources:
  requests:
    cpu: 100m
    memory: 128Mi

replicaCount: 6
ingress:
  enabled: true
  hosts:
    - shop.example.com
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    memory: 1Gi

helm upgrade --install orders ./orders \
  --namespace shop-prod --create-namespace \
  -f values-prod.yaml \
  --set image.tag="$GIT_SHA" \
  --atomic --timeout 5m

This is "build once, promote everywhere" for Kubernetes: the same chart and the same image move through environments, and only a values file and an image tag change. --atomic waits for the resources to become ready and rolls the release back automatically if they do not within the timeout, which is what you want in a pipeline. Maps merge key by key, but lists are replaced whole, so an environment file that sets hosts replaces the default list entirely.

Operating releases

helm list -A                                   # every release in every namespace
helm status orders -n shop-prod
helm history orders -n shop-prod               # revisions, with status and chart version
helm get values orders -n shop-prod            # the values this release was given
helm get manifest orders -n shop-prod          # the exact YAML Helm applied

helm rollback orders 7 -n shop-prod --wait     # back to revision 7 (creates a new revision)
helm uninstall orders -n shop-prod

A rollback re-applies the manifests of an earlier revision. It does not undo what happened outside Kubernetes: a database migration that ran during the upgrade stays applied. That is one reason schema changes should be backwards compatible, a theme that returns in the SRE track's module on safe releases.

Helm only knows about changes made through Helm. If someone runs kubectl edit on a resource that a release owns, the cluster drifts from what Helm recorded, and the next upgrade may overwrite the change or behave unexpectedly. Make every change through the chart. The GitOps module takes that discipline to its conclusion.

Test before it reaches a cluster

Template bugs are cheap to catch locally and expensive to catch in production. Build these commands into CI for every chart change.

helm lint ./orders -f values-prod.yaml              # structural problems and bad practice
helm template orders ./orders -f values-prod.yaml   # render to stdout; no cluster needed
helm template orders ./orders -f values-prod.yaml | kubectl apply --dry-run=server -f -

helm upgrade --install orders ./orders -f values-prod.yaml --dry-run --debug

helm dependency update ./orders                     # fetch subcharts listed in Chart.yaml
helm package ./orders                               # orders-0.3.1.tgz
helm push orders-0.3.1.tgz oci://ghcr.io/acme/charts

helm template piped into a server-side dry run is the most valuable of these: the API server validates the rendered objects against the real schemas without creating anything. Render once per environment file, because a template can be valid with one set of values and broken with another. The helm-diff plugin adds helm diff upgrade, which shows exactly what an upgrade would change and is well worth adding to pull-request checks.