Safe releases

Change is the main source of risk

If change causes most incidents, there are two possible responses. The traditional one is to change less: release monthly, behind a change advisory board, in a maintenance window. It fails for a reason the DORA module explained. Infrequent releases are large, a large release contains hundreds of changes, something in it will break, and finding which of the hundreds is responsible takes hours. Caution of that kind produces exactly the big, risky events it was meant to avoid.

The SRE response is to keep changing, and to make each change safe by construction. Four properties do the work.

• Small. A release containing one change has one suspect. It is easy to review, easy to reason about, and easy to revert.
• Gradual. Expose the change to a small share of traffic first. If it is bad, most users never see it, and the error budget barely notices.
• Observable. You can tell, from metrics split by version, whether the new code is behaving worse than the old.
• Reversible. Undoing the change is quick, tested, and needs nobody's heroics.

Treat every kind of change this way, not only code. Configuration pushes, feature flag changes, infrastructure changes and data migrations cause at least as many outages as deploys do, and they often bypass the pipeline and its safeguards entirely. A configuration value that takes effect globally and instantly is one of the most dangerous things in a modern system.

Rolling, blue-green, canary

A Kubernetes rolling update is controlled by two settings. maxUnavailable is how many pods may be missing during the update, and maxSurge is how many extra may be created. Setting maxUnavailable: 0 means capacity never drops below the desired count, which matters if you have sized your fleet as tightly as the capacity module warned against.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: shop
spec:
  replicas: 12
  minReadySeconds: 30            # a pod must stay ready this long before it counts
  progressDeadlineSeconds: 600   # mark the rollout failed if it stalls
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: ghcr.io/acme/checkout:1.9.0
          readinessProbe:
            httpGet:
              path: /ready
              port: 8000

The weakness of a plain rolling update is what it checks. It waits for each new pod to pass its readiness probe, and nothing else. A version that starts cleanly and then returns errors for 5% of requests, or is twice as slow, passes readiness and rolls out to everyone. Readiness tells you that the process is up, not that the release is good. For that you need to compare the new version's SLIs with the old, which is what a canary does.

Progressive delivery

A canary release is named after the birds once carried into coal mines as an early warning. A small share of real production traffic goes to the new version while the rest stays on the old. Both are measured on the same SLIs at the same time, under the same conditions. If the canary is no worse, its share grows. If it is worse, traffic returns to the old version automatically.

deploy canary
  -> 1% of traffic     bake 10 min   compare error ratio and p99 with the baseline
  -> 5%                bake 10 min   compare
  -> 25%               bake 15 min   compare
  -> 50%               bake 15 min   compare
  -> 100%              keep the old version available for quick rollback

any comparison fails -> traffic back to 0% automatically, release marked failed

• Compare against a baseline, not a fixed threshold. "Error ratio under 1%" passes a canary that is five times worse than the old version on a quiet day. "No worse than the current version, measured now" does not. Ideally the baseline is a freshly started set of old-version pods, so that cold-start effects cancel out.
• Bake long enough to see the problem. Memory leaks, cache effects and hourly jobs do not show in two minutes. Enough requests must flow through the canary for the comparison to mean something, which at 1% of a low-traffic service can take a long time.
• Use the SLIs from your SLOs, split by a version label. If your metrics cannot be split by version, fix that first. Add saturation, such as CPU and memory per pod, and watch downstream dependencies too.
• Order the stages by blast radius: internal users or staff first, then one region or cell, then a small percentage, then everyone. Multi-region services should release one region at a time and never all at once.
• Make the decision automatic. A human watching graphs at each stage is slow and inconsistent, and will stop looking.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
  namespace: shop
spec:
  replicas: 12
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: ghcr.io/acme/checkout:1.9.0
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: {duration: 10m}
        - setWeight: 25
        - pause: {duration: 15m}
        - setWeight: 50
        - pause: {duration: 15m}

Argo Rollouts and Flagger both implement this on Kubernetes, and both can run automated analysis between steps, querying Prometheus and aborting the rollout when a metric fails. Without a service mesh or an ingress that can split traffic, the weight is approximated by the ratio of canary pods to stable pods, so a precise 1% needs real traffic routing.

Feature flags: deploy is not release

Deploying is putting new code on servers. Releasing is exposing new behaviour to users. A feature flag separates the two: the code ships switched off, and is turned on later, at runtime, for whoever you choose, without another deploy. This is what makes trunk-based development workable, because unfinished work can be merged daily and stay dark.

def checkout(user, basket):
    if flags.is_enabled("new_pricing_engine", user=user, default=False):
        try:
            return new_pricing_checkout(user, basket)
        except Exception:
            metrics.increment("new_pricing_engine.fallback")
            # fall through to the proven path
    return legacy_checkout(user, basket)

For reliability, the kill switch is the valuable one. Turning a flag off takes seconds, affects only the feature in question, and needs no build, which makes it the fastest mitigation available, faster than any rollback. Build one into every risky new feature and every expensive optional one, and list them in the runbook.

• Flags are production changes. A flag flip can take a site down as thoroughly as a deploy. Roll flag changes out gradually, record who changed what, and show flag changes as annotations on dashboards, next to deploys.
• Default to safe. If the flag service is unreachable, the code must fall back to a sensible default, normally the old behaviour. The flag system must never be a hard dependency of the request path.
• Flags are debt. Each one doubles the number of code paths, and old flags combine in ways nobody tests. Give every release flag an owner and an expiry date, and delete the flag and the dead code once the rollout is complete.
• Test both states, and the combinations that matter. A well-known trading firm lost hundreds of millions of dollars in under an hour when a reused flag activated long-dead code on one server that had missed a deploy.

Rollback, and the changes that cannot be rolled back

When a release goes wrong, the default is to roll back first and understand later, as the incident response module insisted. Rolling forward, meaning writing and shipping a fix, is slower, is done under pressure, and risks a second bad release on top of the first. Roll forward only when rolling back is impossible or clearly more dangerous.

kubectl rollout status deployment/checkout -n shop --timeout=5m
kubectl rollout history deployment/checkout -n shop
kubectl rollout undo deployment/checkout -n shop                    # previous revision
kubectl rollout undo deployment/checkout -n shop --to-revision=41
kubectl rollout pause deployment/checkout -n shop                   # stop a rollout midway

kubectl argo rollouts get rollout checkout -n shop --watch          # Argo Rollouts plugin
kubectl argo rollouts abort checkout -n shop                        # back to the stable version
kubectl argo rollouts promote checkout -n shop                      # continue past a pause

A rollback is only useful if it is fast, and it is only fast if it has been practised. Time it. If it takes twenty minutes because the old image has to be rebuilt, fix that: keep previous artifacts in the registry, and deploy by immutable tag, as the artifacts module described. In a GitOps setup the rollback is a git revert, and the agent applies it.

The hard part is state. Code can be rolled back, but data cannot. If version 2 has written records in a new format, or dropped a column, then rolling the code back to version 1 leaves it facing a database it does not understand. The answer is to make every schema change backwards compatible, in phases, using expand and contract, also called parallel change.

1. EXPAND    Add the new column full_name, nullable. Old code is unaffected.
2. DUAL WRITE Deploy code that writes to BOTH columns and still reads the old one.
3. BACKFILL  Copy existing data into full_name, in small batches, off-peak.
4. SWITCH    Deploy code that reads full_name (still writing both).
5. VERIFY    Run for a while. Rollback to step 2 or 3 is still safe.
6. CONTRACT  Deploy code that ignores customer_name. Then, separately and much
             later, drop the column.

Every step is its own release, and each can be rolled back by itself.

• Never couple a schema change to the code that needs it in one release. During a rolling update both versions run at once, so the schema must suit both.
• Additive changes are safe: a new table, a new nullable column, a new index built concurrently. Destructive changes are dangerous: dropping or renaming a column, changing a type, adding a NOT NULL constraint without a default.
• Watch for locks. On a large table, an innocent-looking ALTER TABLE can take a lock that blocks all traffic for minutes. Learn which operations your database can perform online, and use its tooling for the rest.
• The same reasoning applies to APIs and message formats. Consumers and producers are deployed at different times, so add fields without removing any, tolerate fields you do not recognise, and version anything that must break.

Putting it into the pipeline

Safe release practice is the delivery pipeline from the DevOps track with reliability built in. A mature flow looks like this.

1. A small change is merged to the trunk behind a flag, after review and automated tests.
2. The artifact is built once, scanned and signed, and promoted unchanged.
3. It deploys to staging, where smoke tests and a short load test run.
4. It deploys to production as a canary. Automated analysis compares it with the baseline at each step and aborts on regression.
5. Deploy and flag events appear as annotations on every dashboard.
6. The feature is released by flag, to staff first, then to a growing share of users, again watching the SLIs.
7. The flag and the old code path are removed.

The error budget is the governor of this machine. With budget in hand, releases flow continuously. When the budget is exhausted, the policy from the SLO module slows feature releases until reliability recovers. Release freezes before peak events are a legitimate use of the same idea, but keep them short: a long freeze builds up a large batch of changes, and the release that follows the freeze is the riskiest of the year.