Kubernetes core

Desired state and the reconciliation loop

You do not tell Kubernetes what to do; you tell it what you want, and it works continuously to make reality match. You submit a manifest, a YAML description of an object, to the API server, which stores it in etcd. Controllers watch for differences between desired and actual state and act to close the gap. If a node dies and takes two of your three replicas with it, the controller notices "want 3, have 1" and starts two more elsewhere. Nobody gets paged.

A cluster has a control plane (API server, scheduler, controller manager, etcd) and worker nodes, each running a kubelet that starts containers and reports back. The scheduler decides which node each new pod runs on. Namespaces divide a cluster into named scopes for teams or environments.

kubectl config get-contexts             # which cluster am I talking to?
kubectl get nodes
kubectl get pods -n shop -o wide        # pods in one namespace, with node and IP
kubectl get all -n shop
kubectl apply -f k8s/                   # create or update everything in a directory
kubectl diff -f k8s/                    # what would apply change?

Pods and Deployments

A Pod is the smallest unit: one or more containers that share a network address and can share volumes. Most pods hold one container. Pods are disposable and get a new IP each time, so you almost never create them directly. You create a Deployment, which manages a ReplicaSet, which keeps the requested number of identical pods running.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
  namespace: shop
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders
  template:
    metadata:
      labels:
        app: orders
    spec:
      containers:
        - name: orders
          image: ghcr.io/acme/orders-api:1.4.2
          ports:
            - containerPort: 8000
          envFrom:
            - configMapRef:
                name: orders-config
            - secretRef:
                name: orders-secrets

Labels are the glue. The Deployment's selector finds its pods by label, and must match the labels in the pod template. When you change the template, typically the image tag, the Deployment performs a rolling update: it starts new pods, waits for them to become ready, and only then removes old ones. If the new version never becomes ready, the rollout stalls with the old pods still serving.

kubectl set image deployment/orders orders=ghcr.io/acme/orders-api:1.4.3 -n shop
kubectl rollout status deployment/orders -n shop
kubectl rollout history deployment/orders -n shop
kubectl rollout undo deployment/orders -n shop      # back to the previous revision
kubectl scale deployment/orders --replicas=5 -n shop

Services and Ingress

Pods come and go, so nothing should talk to a pod IP. A Service provides a stable virtual IP and DNS name, and load-balances across every ready pod matching its selector. Inside the cluster, the Service below is reachable as orders from the same namespace, or as orders.shop.svc.cluster.local from anywhere.

apiVersion: v1
kind: Service
metadata:
  name: orders
  namespace: shop
spec:
  selector:
    app: orders
  ports:
    - port: 80            # the Service's port
      targetPort: 8000    # the container's port

One cloud load balancer per service gets expensive. An Ingress routes HTTP traffic for many services through a single entry point, by hostname and path, and usually terminates TLS. The Ingress object is only a set of rules; an ingress controller such as ingress-nginx or Traefik must be installed to implement them.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: shop
spec:
  ingressClassName: nginx
  tls:
    - hosts: [shop.example.com]
      secretName: shop-tls
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /api/orders
            pathType: Prefix
            backend:
              service:
                name: orders
                port:
                  number: 80

ConfigMaps and Secrets

This is factor 3 of the twelve-factor app, implemented by the platform. A ConfigMap holds non-sensitive settings and a Secret holds sensitive ones. Both can be injected as environment variables, as in the Deployment above, or mounted as files.

apiVersion: v1
kind: ConfigMap
metadata:
  name: orders-config
  namespace: shop
data:
  LOG_LEVEL: info
  PAYMENTS_URL: http://payments.shop.svc.cluster.local
---
apiVersion: v1
kind: Secret
metadata:
  name: orders-secrets
  namespace: shop
type: Opaque
stringData:
  DATABASE_URL: postgres://orders:CHANGE_ME@db.shop.svc.cluster.local:5432/orders

Pods read environment variables once, at start. Changing a ConfigMap does not restart anything, so trigger a new rollout with kubectl rollout restart deployment/orders. Tools such as Helm and Kustomize can automate that by changing the pod template whenever the config changes.

Probes, requests and limits

Kubernetes can only manage your app well if it knows whether the app is healthy and how much it needs. Probes answer the first question.

readinessProbe:
  httpGet:
    path: /ready
    port: 8000
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /healthz
    port: 8000
  periodSeconds: 10
  failureThreshold: 3
resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    memory: 512Mi

Readiness is what makes rolling updates safe, and it may check dependencies the pod needs in order to serve. Liveness should check only the process itself. A liveness probe that fails when the database is down makes Kubernetes restart every pod during a database outage, turning one problem into two.

Requests are what the scheduler reserves: a pod is only placed on a node with that much unallocated capacity. Limits are enforced ceilings. CPU is measured in millicores (250m is a quarter of a core) and is compressible: exceed the limit and the container is throttled. Memory is not: exceed the limit and the container is killed with OOMKilled. Always set requests; without them the scheduler is guessing and one greedy pod can starve its neighbours.

The debugging loop

Almost every problem yields to the same four commands, in this order.

kubectl get pods -n shop                        # 1. what state is it in?
kubectl describe pod orders-7d9f8c6b5-x2k4q -n shop   # 2. read Events at the bottom
kubectl logs orders-7d9f8c6b5-x2k4q -n shop --previous  # 3. logs of the crashed container
kubectl get events -n shop --sort-by=.lastTimestamp     # 4. everything that happened

kubectl exec -it deploy/orders -n shop -- sh    # look around inside
kubectl port-forward svc/orders 8080:80 -n shop # reach a Service from your laptop
kubectl top pods -n shop                        # live CPU and memory (needs metrics-server)

Use kubectl explain deployment.spec.strategy to read the documentation for any field without leaving the terminal, and kubectl get deployment orders -o yaml to see the object as the cluster actually holds it, defaults included.