Progressive delivery with automated analysis

The Rollouts controller runs in the cluster, managed by Argo CD like everything else, and its metrics are scraped by Prometheus.

1. 1.1
   Confirm the GitOps cluster is up and the pieces this project leans on are healthy: ingress-nginx, Argo CD, Prometheus and the orders-api dev application.

   bashCopy

   kubectl config use-context kind-gitops
   kubectl -n ingress-nginx get pods
   argocd app list
   kubectl -n monitoring get svc | grep prometheus
   curl -s http://orders.localtest.me/health

   Note the Prometheus Service name — with the chart released as monitoring it is monitoring-kube-prometheus-prometheus on port 9090. The AnalysisTemplate needs that address.
2. 1.2
   In the config repository, add a platform folder for Argo Rollouts. Kustomize pulls the pinned upstream install manifest and adds a ServiceMonitor so the controller's own metrics (rollout phase, analysis results) land in Prometheus.

   yamlCopy

   # platform/argo-rollouts/kustomization.yaml
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   namespace: argo-rollouts
   resources:
     - https://github.com/argoproj/argo-rollouts/releases/download/v1.8.3/install.yaml
     - servicemonitor.yaml
   
   # platform/argo-rollouts/servicemonitor.yaml
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: argo-rollouts
     labels: {release: monitoring}
   spec:
     selector:
       matchLabels: {app.kubernetes.io/name: argo-rollouts-metrics}
     endpoints:
       - port: metrics
         interval: 30s

   Check the Argo Rollouts releases page for the current tag and pin that. The release: monitoring label is what the kube-prometheus-stack from the GitOps project uses to pick up ServiceMonitors.
3. 1.3
   Add the Argo CD Application that points at the folder, with CreateNamespace and ServerSideApply (the CRDs are large; server-side apply avoids the annotation size limit).

   yamlCopy

   # argocd/applications/argo-rollouts.yaml
   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: argo-rollouts
     namespace: argocd
   spec:
     project: platform
     source:
       repoURL: https://github.com/YOUR_GITHUB_USER/zero-to-prod-config
       targetRevision: main
       path: platform/argo-rollouts
     destination:
       server: https://kubernetes.default.svc
       namespace: argo-rollouts
     syncPolicy:
       automated: {prune: true, selfHeal: true}
       syncOptions: [CreateNamespace=true, ServerSideApply=true]

4. 1.4
   Commit, let the app-of-apps pick it up, and install the kubectl plugin on your laptop.

   bashCopy

   cd zero-to-prod-config
   git switch -c feat/argo-rollouts
   git add platform/argo-rollouts argocd/applications/argo-rollouts.yaml
   git commit -m "platform: Argo Rollouts controller + metrics"
   git push -u origin feat/argo-rollouts && gh pr create --fill && gh pr merge --squash --delete-branch
   git switch main && git pull
   argocd app wait argo-rollouts --sync --health --timeout 300
   kubectl -n argo-rollouts get pods
   brew install argoproj/tap/kubectl-argo-rollouts   # Linux: download the release binary
   kubectl argo rollouts version

The chart can render either a plain Deployment or a Rollout with a canary strategy; the Rollout shifts traffic through the existing ingress in weighted steps.

1. 2.1
   In the app repository, add the rollout settings to chart/values.yaml. The steps are the release plan: weight, then wait for analysis, then more weight. Keep the Deployment as the default so the chart still works anywhere without Argo Rollouts.

   yamlCopy

   # chart/values.yaml (append)
   rollout:
     enabled: false
     steps:
       - setWeight: 10
       - pause: {duration: 2m}
       - setWeight: 25
       - pause: {duration: 2m}
       - setWeight: 50
       - pause: {duration: 3m}
     analysis:
       enabled: true
       startingStep: 1          # begin judging as soon as the first 10 % is live
       prometheus: http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090
       minSuccessRate: "0.99"
       maxLatencyRatio: "1.5"   # canary p99 may be at most 1.5x stable p99

2. 2.2
   Guard the existing Deployment template so it renders only when rollouts are off. Wrap the whole of chart/templates/deployment.yaml.

   helmCopy

   {{- if not .Values.rollout.enabled }}
   apiVersion: apps/v1
   kind: Deployment
   # ... the existing template, unchanged ...
   {{- end }}

3. 2.3
   Write chart/templates/rollout.yaml. The pod template is the same as the Deployment's; what differs is strategy.canary: a stable and a canary Service, ingress-nginx traffic routing against the chart's Ingress, and the steps from values. Background analysis runs the template continuously from startingStep onward.

   helmCopy

   {{- if .Values.rollout.enabled }}
   apiVersion: argoproj.io/v1alpha1
   kind: Rollout
   metadata:
     name: {{ .Release.Name }}
     labels:
       app.kubernetes.io/name: {{ .Chart.Name }}
       app.kubernetes.io/instance: {{ .Release.Name }}
   spec:
     replicas: {{ .Values.replicaCount }}
     revisionHistoryLimit: 3
     selector:
       matchLabels:
         app.kubernetes.io/instance: {{ .Release.Name }}
     template:
       metadata:
         labels:
           app.kubernetes.io/name: {{ .Chart.Name }}
           app.kubernetes.io/instance: {{ .Release.Name }}
       spec:
         containers:
           - name: app
             image: "{{ .Values.image.repository }}{{ if .Values.image.digest }}@{{ .Values.image.digest }}{{ else }}:{{ .Values.image.tag }}{{ end }}"
             ports:
               - name: http
                 containerPort: {{ .Values.service.targetPort }}
             env:
               {{- range $k, $v := .Values.env }}
               - name: {{ $k }}
                 value: {{ $v | quote }}
               {{- end }}
             readinessProbe:
               httpGet: {path: /health, port: http}
               periodSeconds: 5
             resources: {{- toYaml .Values.resources | nindent 12 }}
     strategy:
       canary:
         stableService: {{ .Release.Name }}
         canaryService: {{ .Release.Name }}-canary
         trafficRouting:
           nginx:
             stableIngress: {{ .Release.Name }}
         {{- if .Values.rollout.analysis.enabled }}
         analysis:
           templates:
             - templateName: {{ .Release.Name }}-canary-vs-stable
           startingStep: {{ .Values.rollout.analysis.startingStep }}
           args:
             - name: canary-hash
               valueFrom: {podTemplateHashValue: Latest}
             - name: stable-hash
               valueFrom: {podTemplateHashValue: Stable}
         {{- end }}
         steps: {{- toYaml .Values.rollout.steps | nindent 8 }}
   {{- end }}

   stableIngress must be the name of the chart's Ingress and that Ingress must point at the stable Service. Argo Rollouts creates a second Ingress next to it with the nginx.ingress.kubernetes.io/canary-weight annotation and moves the number as the steps advance — no change to ingress-nginx itself.
4. 2.4
   Add the canary Service. It is identical to the stable one; the controller rewrites each Service's selector with the pod-template hash it should point at. Because both carry the chart's labels, the existing ServiceMonitor scrapes canary and stable pods alike — that is what makes the comparison possible.

   helmCopy

   {{- if .Values.rollout.enabled }}
   apiVersion: v1
   kind: Service
   metadata:
     name: {{ .Release.Name }}-canary
     labels:
       app.kubernetes.io/name: {{ .Chart.Name }}
       app.kubernetes.io/instance: {{ .Release.Name }}
   spec:
     selector:
       app.kubernetes.io/instance: {{ .Release.Name }}
     ports:
       - name: http
         port: {{ .Values.service.port }}
         targetPort: http
   {{- end }}

5. 2.5
   Render both modes locally to be sure the chart is still valid, then bump the chart version.

   bashCopy

   cd zero-to-prod
   helm lint chart
   helm template orders-api chart | grep -c '^kind: Deployment'                     # 1
   helm template orders-api chart --set rollout.enabled=true | grep -E '^kind: (Rollout|Deployment|Service)'
   sed -i.bak 's/^version: 0.1.0/version: 0.2.0/' chart/Chart.yaml && rm chart/Chart.yaml.bak

Two metrics decide every step: the canary's success rate must stay above 99 %, and its p99 latency may not exceed 1.5× the stable version's. Both are PromQL against the pods of each revision.

1. 3.1
   Write chart/templates/analysis.yaml. Each metric is queried every 30 s; failureLimit: 2 means two bad samples abort the rollout, inconclusiveLimit pauses it when the query returns nothing (no traffic yet). Pods are selected by name, because ReplicaSets created by a Rollout are named <rollout>-<pod-template-hash>.

   helmCopy

   {{- if and .Values.rollout.enabled .Values.rollout.analysis.enabled }}
   apiVersion: argoproj.io/v1alpha1
   kind: AnalysisTemplate
   metadata:
     name: {{ .Release.Name }}-canary-vs-stable
   spec:
     args:
       - name: canary-hash
       - name: stable-hash
     metrics:
       - name: success-rate
         interval: 30s
         failureLimit: 2
         inconclusiveLimit: 3
         successCondition: len(result) > 0 && result[0] >= {{ .Values.rollout.analysis.minSuccessRate }}
         provider:
           prometheus:
             address: {{ .Values.rollout.analysis.prometheus }}
             query: |
               sum(rate(http_requests_total{pod=~"{{ .Release.Name }}-{{ "{{args.canary-hash}}" }}-.*",status!~"5.."}[2m]))
               /
               sum(rate(http_requests_total{pod=~"{{ .Release.Name }}-{{ "{{args.canary-hash}}" }}-.*"}[2m]))
       - name: latency-vs-stable
         interval: 30s
         failureLimit: 2
         inconclusiveLimit: 3
         successCondition: len(result) > 0 && result[0] <= {{ .Values.rollout.analysis.maxLatencyRatio }}
         provider:
           prometheus:
             address: {{ .Values.rollout.analysis.prometheus }}
             query: |
               histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{pod=~"{{ .Release.Name }}-{{ "{{args.canary-hash}}" }}-.*"}[2m])))
               /
               histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{pod=~"{{ .Release.Name }}-{{ "{{args.stable-hash}}" }}-.*"}[2m])))
   {{- end }}

   The odd-looking {{ "{{args.canary-hash}}" }} is Helm printing a literal {{args.canary-hash}} so that Argo Rollouts, not Helm, substitutes it. Render the template and read the output once; it should contain the plain {{args.…}} placeholders.
2. 3.2
   Render and read the AnalysisTemplate, then run both queries by hand in Prometheus with the current stable hash to confirm they return numbers. A query that returns nothing here will return nothing during a rollout.

   bashCopy

   helm template orders-api chart --set rollout.enabled=true | sed -n '/kind: AnalysisTemplate/,/^---/p'
   kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 &
   sleep 2
   curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=sum(rate(http_requests_total{pod=~"orders-api-.*",status!~"5.."}[2m])) / sum(rate(http_requests_total{pod=~"orders-api-.*"}[2m]))' | python3 -m json.tool | head -20
   kill %1

   If the result is empty, generate traffic first (hey -z 3m -q 5 http://orders.localtest.me/orders/1) and check the ServiceMonitor target is up under Status → Targets.
3. 3.3
   Commit the chart, merge, and let the release workflow from the GitOps project publish chart 0.2.x and bump the dev overlay. The app image is unchanged; only the chart moves.

   bashCopy

   git switch -c feat/canary-rollout
   git add chart
   git commit -m "chart: optional Rollout with weighted canary and Prometheus analysis"
   git push -u origin feat/canary-rollout && gh pr create --fill
   gh pr checks --watch && gh pr merge --squash --delete-branch
   git switch main && git pull
   gh run watch

4. 3.4
   In the config repository, switch the dev environment to rollouts. Turning rollout.enabled on replaces the Deployment with a Rollout; Argo CD handles the swap as one sync. Argo CD's built-in health check understands Rollouts, so the app shows *Progressing* during a canary and *Degraded* if one is aborted.

   bashCopy

   cd ../zero-to-prod-config && git pull
   git switch -c feat/dev-canary
   cat >> apps/orders-api/base/values.yaml <<'EOF'
   rollout:
     enabled: true
   EOF
   git add apps/orders-api/base/values.yaml
   git commit -m "orders-api: canary rollouts in all environments"
   git push -u origin feat/dev-canary && gh pr create --fill && gh pr merge --squash --delete-branch
   git switch main && git pull
   argocd app wait orders-api-dev --sync --health --timeout 300
   kubectl -n orders-dev get rollout,svc,ingress

   The first sync creates the Rollout fresh, so there is no canary yet — the controller treats the initial revision as stable. Expected: one Rollout, two Services (orders-api, orders-api-canary), two Ingresses (the chart's and orders-api-orders-api-canary).

With steady traffic flowing, a harmless change walks through 10 → 25 → 50 → 100 % with every analysis measurement green, and you can see each step in the CLI, the dashboard and Prometheus.

1. 4.1
   Start a load generator in its own terminal and leave it running for the rest of the project. Twenty requests per second is enough for the 2-minute rate windows to be meaningful at a 10 % canary weight.

   bashCopy

   hey -z 60m -q 20 -c 2 -disable-keepalive http://orders.localtest.me/orders/42

   -disable-keepalive matters: ingress-nginx weights new connections, so a client that reuses one connection would stick to whichever version it hit first and the canary would see no traffic.
2. 4.2
   In a second terminal, watch the Rollout, and in a third open the Rollouts dashboard.

   bashCopy

   kubectl argo rollouts -n orders-dev get rollout orders-api --watch
   # third terminal:
   kubectl argo rollouts dashboard   # http://localhost:3100 — pick namespace orders-dev

3. 4.3
   Make a harmless change through the normal flow: a new greeting in the app repo, merged, built, and bumped into dev by the release workflow's pull request. Merge that bump. The digest change is the pod-template change that starts the canary.

   bashCopy

   cd ../zero-to-prod && git switch -c feat/canary-greeting
   sed -i.bak 's/hello from the pipeline/hello from the canary/' app/main.py && rm app/main.py.bak
   git commit -am "feat: greeting for the first canary" && git push -u origin feat/canary-greeting
   gh pr create --fill && gh pr checks --watch && gh pr merge --squash --delete-branch
   git switch main && git pull && gh run watch
   cd ../zero-to-prod-config && gh pr list      # the bump/dev-… PR from the release bot
   gh pr merge --squash --delete-branch $(gh pr list --search 'bump/dev' --json number -q '.[0].number')

4. 4.4
   Follow the rollout. In the watch terminal the status goes Paused at SetWeight: 10 with an AnalysisRun Running; the canary Ingress annotation shows the weight; and the greeting alternates roughly one in ten. After the 2-minute pause with passing analysis it moves to 25, then 50, then finishes.

   bashCopy

   kubectl -n orders-dev get ingress orders-api-orders-api-canary -o jsonpath='{.metadata.annotations.nginx\.ingress\.kubernetes\.io/canary-weight}{"\n"}'
   for i in $(seq 1 20); do curl -s http://orders.localtest.me/ | grep -o 'hello from [a-z ]*'; done | sort | uniq -c
   kubectl -n orders-dev get analysisrun
   kubectl -n orders-dev get analysisrun -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.phase}{" success-rate="}{.status.metricResults[0].successful}{" latency="}{.status.metricResults[1].successful}{"\n"}{end}'

   Roughly eight minutes end to end. If the AnalysisRun stays Running with measurements but the rollout does not advance, it is simply inside a pause — read the step index in the watch output.
5. 4.5
   Read the measurements the controller took, so the verdict is not a black box. Each measurement is a timestamp, the raw query result and a phase.

   bashCopy

   RUN=$(kubectl -n orders-dev get analysisrun --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
   kubectl -n orders-dev get analysisrun "$RUN" -o jsonpath='{range .status.metricResults[*]}{.name}{": "}{.phase}{" measurements="}{.count}{" failed="}{.failed}{"\n"}{range .measurements[*]}{"  "}{.startedAt}{"  "}{.value}{"  "}{.phase}{"\n"}{end}{end}'

6. 4.6
   Record the release in the config repository's README table: date, digest, duration, outcome. A release log you can read later is part of the practice, not decoration.

   bashCopy

   cd ../zero-to-prod-config
   DIG=$(grep -o 'sha256:[0-9a-f]*' apps/orders-api/dev/kustomization.yaml)
   printf '| %s | dev | %s | canary 10/25/50/100, analysis green, ~8 min | promoted |\n' "$(date -u +%F)" "${DIG:7:12}" >> RELEASES.md
   git add RELEASES.md && git commit -m "releases: first canary" && git push

A version with a 5 % error rate reaches 10 % of traffic and no further: the success-rate metric fails twice, the rollout aborts, traffic returns to stable, and you can see all of it.

1. 5.1
   The bad release is a configuration change, which is deliberately realistic: most bad deploys are not bad code. Patch the dev overlay to set CHAOS_ERROR_RATE=0.05 on the Rollout's pod template. A pod-template change is a new revision, so it goes through the canary like an image change would.

   yamlCopy

   # apps/orders-api/dev/kustomization.yaml — add to patches:
     - target: {kind: Rollout, name: orders-api}
       patch: |
         - op: add
           path: /spec/template/spec/containers/0/env/-
           value: {name: CHAOS_ERROR_RATE, value: "0.05"}

   If the dev overlay already patches /spec/replicas on a Deployment, change that target to kind: Rollout too — the Deployment no longer exists in this environment.
2. 5.2
   Merge it and watch. Within a minute the canary is at 10 %; within two to three minutes the success-rate measurements read about 0.95, the second failure trips failureLimit, and the Rollout aborts: weight back to 0, canary pods scaled down, status Degraded.

   bashCopy

   git switch -c chaos/bad-release
   git commit -am "dev: 5% error injection (bad release drill)" && git push -u origin chaos/bad-release
   gh pr create --fill && gh pr merge --squash --delete-branch && git switch main && git pull
   kubectl argo rollouts -n orders-dev get rollout orders-api --watch

3. 5.3
   Inspect what happened: the AnalysisRun's failed measurements, the Rollout's events, and Argo CD's view of the application, which is now Degraded because the Rollout is.

   bashCopy

   RUN=$(kubectl -n orders-dev get analysisrun --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
   kubectl -n orders-dev get analysisrun "$RUN" -o jsonpath='{.status.phase}{" "}{.status.message}{"\n"}'
   kubectl -n orders-dev get analysisrun "$RUN" -o jsonpath='{range .status.metricResults[0].measurements[*]}{.startedAt}{"  "}{.value}{"  "}{.phase}{"\n"}{end}'
   kubectl -n orders-dev get events --field-selector involvedObject.name=orders-api --sort-by=.lastTimestamp | tail -15
   argocd app get orders-api-dev | grep -E 'Health|Sync'
   for i in $(seq 1 50); do curl -s -o /dev/null -w '%{http_code}\n' http://orders.localtest.me/orders/42; done | sort | uniq -c

   The last command should show only 200s: users are back on stable. During the canary they saw about 0.5 % errors (5 % of 10 %) for two or three minutes — that is the cost of the experiment, and the reason the first step is 10 % and not 50 %.
4. 5.4
   Fix forward the way you would at work: revert the bad change in Git. Argo CD syncs the revert, which is a *new* revision as far as the Rollout is concerned — but because its pod template equals the stable one, the controller recognises it and completes immediately with no canary.

   bashCopy

   git revert --no-edit HEAD && git push
   argocd app wait orders-api-dev --sync --health --timeout 300
   kubectl argo rollouts -n orders-dev get rollout orders-api
   printf '| %s | dev | chaos 5%% | aborted at 10%% after 2 failed success-rate measurements (~3 min) | rolled back automatically |\n' "$(date -u +%F)" >> RELEASES.md
   git add RELEASES.md && git commit -m "releases: bad release drill" && git push

   Argo Rollouts also supports kubectl argo rollouts abort and undo for hand-driven cases; in a GitOps setup the revert is the honest record of what happened.

During any rollout, one Grafana screen shows canary and stable success rate and p99 side by side with the current weight, and an aborted rollout raises an alert through the existing Alertmanager routes.

1. 6.1
   Build the dashboard in Grafana's UI with these four panels, all using the pod-name pattern so they split by revision. Use sum by (pod) variants first to see individual pods, then the aggregated form shown here.

   promqlCopy

   # Success rate by revision (one line per ReplicaSet hash)
   sum by (rs) (label_replace(rate(http_requests_total{namespace="orders-dev",status!~"5.."}[2m]), "rs", "$1", "pod", "(orders-api-[a-z0-9]+)-.*"))
     / sum by (rs) (label_replace(rate(http_requests_total{namespace="orders-dev"}[2m]), "rs", "$1", "pod", "(orders-api-[a-z0-9]+)-.*"))
   
   # p99 latency by revision
   histogram_quantile(0.99, sum by (rs, le) (label_replace(rate(http_request_duration_seconds_bucket{namespace="orders-dev"}[2m]), "rs", "$1", "pod", "(orders-api-[a-z0-9]+)-.*")))
   
   # Requests per second by revision (shows the weight shifting)
   sum by (rs) (label_replace(rate(http_requests_total{namespace="orders-dev"}[1m]), "rs", "$1", "pod", "(orders-api-[a-z0-9]+)-.*"))
   
   # Rollout phase from the controller's own metrics (stat panel)
   rollout_info{namespace="orders-dev", name="orders-api"}

   label_replace derives a rs label from the pod name, which is the same trick the AnalysisTemplate relies on. Show the phase label of rollout_info as the stat panel's text.
2. 6.2
   Export the dashboard as JSON and commit it to the config repo as a ConfigMap; the kube-prometheus-stack sidecar loads any ConfigMap labelled grafana_dashboard: "1". Add it to the monitoring kustomization's resources.

   bashCopy

   mkdir -p platform/monitoring/dashboards
   # In Grafana: dashboard → Share → Export → Save to file → platform/monitoring/dashboards/releases.json
   kubectl create configmap dashboard-releases -n monitoring \
     --from-file=releases.json=platform/monitoring/dashboards/releases.json \
     --dry-run=client -o yaml > platform/monitoring/dashboard-releases.yaml
   sed -i.bak 's/^metadata:/metadata:\n  labels: {grafana_dashboard: "1"}/' platform/monitoring/dashboard-releases.yaml && rm platform/monitoring/dashboard-releases.yaml.bak
   grep -n 'grafana_dashboard' platform/monitoring/dashboard-releases.yaml

3. 6.3
   Add a PrometheusRule for aborted rollouts. It joins the SLO project's Alertmanager routing: severity: page reaches the on-call receiver.

   yamlCopy

   # platform/monitoring/rules/rollouts.yaml
   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: rollouts
     namespace: monitoring
     labels: {release: monitoring}
   spec:
     groups:
       - name: rollouts
         rules:
           - alert: RolloutAborted
             expr: rollout_info{phase="Degraded"} == 1
             for: 1m
             labels: {severity: page}
             annotations:
               summary: "Rollout {{ $labels.namespace }}/{{ $labels.name }} was aborted by analysis"
               runbook: "https://github.com/YOUR_GITHUB_USER/zero-to-prod-config/blob/main/RUNBOOK-rollouts.md"
           - alert: RolloutStuck
             expr: rollout_info{phase="Paused"} == 1
             for: 30m
             labels: {severity: ticket}
             annotations:
               summary: "Rollout {{ $labels.namespace }}/{{ $labels.name }} has been paused for 30 minutes"

4. 6.4
   Commit both, sync, and re-run the bad release from the previous phase to see the alert fire and the dashboard split. Then revert again.

   bashCopy

   git switch -c feat/release-dashboard
   git add platform/monitoring
   git commit -m "monitoring: release dashboard and rollout alerts"
   git push -u origin feat/release-dashboard && gh pr create --fill && gh pr merge --squash --delete-branch
   git switch main && git pull && argocd app wait monitoring --sync --health --timeout 300
   git revert --no-edit HEAD~2   # re-applies the chaos patch (the revert of the revert)
   git push && kubectl argo rollouts -n orders-dev get rollout orders-api --watch
   # once Degraded: check Alertmanager at http://alertmanager.localtest.me (or port-forward) for RolloutAborted, then:
   git revert --no-edit HEAD && git push

   HEAD~2 assumes the last three commits on main are: dashboard, revert-of-chaos, chaos. Check git log --oneline -4 first and pick the revert commit's SHA if the order differs.

The prod overlay uses the same chart, the same analysis, but pauses indefinitely at 50 % so promotion is a deliberate act — and the release procedure is written down.

1. 7.1
   Patch the prod Rollout's steps: replace the timed pause after 50 % with an indefinite one. The analysis keeps running in the background while it waits, so a canary that goes bad during the pause still aborts on its own.

   yamlCopy

   # apps/orders-api/prod/kustomization.yaml — add:
   patches:
     - target: {kind: Rollout, name: orders-api}
       patch: |
         - op: replace
           path: /spec/strategy/canary/steps
           value:
             - setWeight: 10
             - pause: {duration: 5m}
             - setWeight: 50
             - pause: {}

2. 7.2
   Promote the good digest to prod through the usual pull request, then watch it stop at 50 % and promote it by hand.

   bashCopy

   git switch -c promote/prod-canary
   DIG=$(grep -o 'sha256:[0-9a-f]*' apps/orders-api/dev/kustomization.yaml)
   sed -i.bak "s#digest: sha256:.*#digest: ${DIG}#" apps/orders-api/prod/kustomization.yaml && rm apps/orders-api/prod/kustomization.yaml.bak
   git add apps/orders-api/prod && git commit -m "prod: promote ${DIG:7:12} with manual gate at 50%"
   git push -u origin promote/prod-canary && gh pr create --fill && gh pr merge --squash --delete-branch
   git switch main && git pull
   kubectl argo rollouts -n orders-prod get rollout orders-api --watch
   # when the step shows 'pause' with no duration and the analysis is green:
   kubectl argo rollouts -n orders-prod promote orders-api

   Point hey at the prod host while this runs, or the analysis will go inconclusive for lack of traffic and pause the rollout before it reaches the gate.
3. 7.3
   Write RUNBOOK-rollouts.md in the config repo: how to read get rollout, what promote, abort and retry do, what the two alerts mean, and the Git revert as the standard rollback. Link the Grafana dashboard.

   bashCopy

   cat > RUNBOOK-rollouts.md <<'EOF'
   # Rollouts runbook
   
   **Watch:** `kubectl argo rollouts -n <ns> get rollout orders-api --watch` — step index, weight, AnalysisRun phase.
   **Dashboard:** Grafana → Releases (canary vs stable success rate and p99, weight, phase).
   
   ## RolloutAborted (page)
   Analysis failed twice. Users are already back on stable. Read the AnalysisRun measurements, find the cause in the
   canary pods' logs, then `git revert` the change in this repo. Do not `retry` without a fix.
   
   ## RolloutStuck (ticket)
   Paused >30 min: either the prod manual gate is waiting for a human, or analysis is Inconclusive (no traffic).
   Gate: `kubectl argo rollouts -n orders-prod promote orders-api`. Inconclusive: check Prometheus targets and traffic.
   
   ## Rollback
   Revert the commit in Git. `kubectl argo rollouts abort` is for emergencies; follow it with the revert so Git matches the cluster.
   EOF
   git add RUNBOOK-rollouts.md && git commit -m "docs: rollouts runbook" && git push

4. 7.4
   Finish by tightening the analysis with what you learned: shorter windows would have caught the bad release faster but with more noise; a stricter ratio would block a slightly slower but correct release. Pick values, change them in chart/values.yaml, and record the reasoning in the release log.

   bashCopy

   cd ../zero-to-prod
   sed -n '/^rollout:/,$p' chart/values.yaml
   # Adjust minSuccessRate / maxLatencyRatio / the pause durations, bump chart version, PR, merge.
   # Then in the config repo:
   #   printf '| %s | tuning | minSuccessRate 0.995, 90s pauses | reason: ... |\n' "$(date -u +%F)" >> RELEASES.md

The AnalysisRun stays Inconclusive and the rollout pauses at 10 %

The query returned no data: the canary pods are not being scraped or have had no traffic in the last two minutes. Check Prometheus → Status → Targets for the orders-api-canary endpoints, confirm the load generator runs with keep-alive disabled, and run the success-rate query by hand with the canary hash from kubectl argo rollouts get rollout.

kubectl argo rollouts get rollout shows Error on the AnalysisRun with connection refused

The Prometheus address in values.yaml does not match the Service name in the monitoring namespace. kubectl -n monitoring get svc and use <service>.<namespace>.svc:9090. The controller runs in argo-rollouts, so a localhost address will never work.

Traffic never reaches the canary (greeting is always the stable one)

Look at the canary Ingress: kubectl -n orders-dev get ingress orders-api-orders-api-canary -o yaml. Missing canary: "true" annotations means stableIngress in the Rollout does not name the chart's Ingress. If the annotations are there, the client is reusing connections — re-run hey with -disable-keepalive and curl without a session.

Argo CD shows the app OutOfSync forever after enabling rollouts

The controller rewrites the two Services' selectors with the pod-template hash, which differs from the chart's rendered selector. Add an ignoreDifferences entry for Service /spec/selector on the Application, or use the Argo CD RespectIgnoreDifferences=true sync option.

The rollout aborts on latency-vs-stable although nothing is slower

With little traffic, p99 from a single canary pod is noisy. Raise the traffic, widen the rate window to [5m], or start the latency metric at a later step (startingStep: 3) once the canary has enough requests to make a quantile meaningful.

After the revert, the Rollout runs a full canary instead of completing immediately

The revert's pod template is not byte-identical to stable — usually an extra label or annotation from a patch. Compare kubectl get rs -n orders-dev -o yaml for the two ReplicaSets; the diff is the field to remove.

• Add a smoke-test Job provider to the AnalysisTemplate that runs your API tests against the canary Service before any traffic is shifted (setWeight: 0 + inline analysis as step one).
• Replace ingress-nginx weighting with header-based routing for internal users first (setHeaderRoute), so staff see the canary before the public does.
• Run the SRE incident drill against an aborted rollout: the page, the timeline, and a postmortem asking why the change was not caught before the canary.
• Feed the SLO project's burn rate into the analysis as a third metric — a canary that burns budget faster than 2× stable aborts even if its absolute error rate is under the threshold.
• Try the same strategy on AWS with the DevOps track's ECS platform using CodeDeploy blue/green with CloudWatch alarms as the analysis, and compare the two mechanisms.