Education › DevOps Engineering › Guided project

GitOps on Kubernetes with Argo CD

Run the zero-to-production API on a local Kubernetes cluster the GitOps way: a Helm chart for the app, a separate config repository that describes what should be running, Argo CD keeping the cluster equal to that repository, Prometheus and Grafana installed the same way, and a release flow where the only path to production is a merged pull request that bumps an image digest. Drift, rollback and environment promotion all become git operations.

Intermediate about 6 hours 5 phases · 25 steps 0 / 25 done
Overview 1. Cluster and Argo CD 2. A Helm chart for the app 3. The config repository and Argo CD Applications 4. The monitoring stack, the same way 5. The release flow, drift and rollback Troubleshooting
What you will have at the end

Two repositories — the app repo with a Helm chart and a CI job that builds, pushes and opens a pull request against the config repo, and the config repo with Kustomize overlays for dev and prod that Argo CD syncs to a kind cluster. Argo CD's UI shows the app, the monitoring stack and their health; a manual kubectl edit is reverted automatically; a bad release is rolled back by reverting a commit; prod only changes after dev has been synced and healthy.

Before you start
  • The zero-to-production project (its image on GHCR is what you deploy), or any container image with a /health and /metrics endpoint
  • The DevOps track's Kubernetes, Helm and GitOps modules
  • Docker Desktop, kind, kubectl, Helm and the GitHub CLI installed; 6 GB RAM free for the cluster
Tools you will install
  • kind — a local Kubernetes cluster in Docker, rebuilt in a minute ↗
  • Helm — packages the app with values per environment ↗
  • Kustomize — environment overlays in the config repo without templating everything ↗
  • Argo CD — continuous reconciliation of the cluster to git, with a UI that shows drift and health ↗
  • kube-prometheus-stack — Prometheus, Alertmanager and Grafana as one chart, deployed by Argo CD like everything else ↗
Repository layout at the end
zero-to-prod/                      # app repo (existing)
├── chart/
│   ├── Chart.yaml
│   ├── values.yaml
│   └── templates/ (deployment, service, ingress, servicemonitor, hpa)
└── .github/workflows/release.yml  # + bump the digest in the config repo via PR

zero-to-prod-config/               # config repo (new): the desired state
├── apps/
│   └── orders-api/
│       ├── base/kustomization.yaml         # HelmChart inflation from the app repo's chart
│       ├── dev/kustomization.yaml          # image digest + dev values
│       └── prod/kustomization.yaml         # image digest + prod values
├── platform/
│   ├── monitoring/                         # kube-prometheus-stack Application
│   └── ingress-nginx/
├── argocd/
│   ├── project.yaml                        # AppProject with allowed repos and namespaces
│   ├── app-of-apps.yaml                    # root Application pointing at apps/ and platform/
│   └── applications/*.yaml
└── README.md

Tick each step as you finish it — your progress is saved in this browser only (back up or restore on the hub). Every code block has a copy button. If something goes wrong, the troubleshooting section at the end covers the usual suspects.

Phase 1

Cluster and Argo CD

A kind cluster with ingress, Argo CD installed and reachable, and the CLI logged in.

  1. Create a kind cluster with port mappings for the ingress controller, so services are reachable at localhost.
    bash
    cat > kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
      - containerPort: 443
        hostPort: 443
  - role: worker
EOF
kind create cluster --name gitops --config kind.yaml
kubectl cluster-info --context kind-gitops
    Check: kubectl get nodes shows two Ready nodes.
  2. Install the ingress-nginx controller for kind (this one is installed by hand; Argo CD will manage everything after it exists), then Argo CD.
    bash
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0/deploy/static/provider/kind/deploy.yaml
kubectl -n ingress-nginx wait --for=condition=ready pod -l app.kubernetes.io/component=controller --timeout=180s

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.13.3/manifests/install.yaml
kubectl -n argocd wait --for=condition=available deploy/argocd-server --timeout=300s
    Argo CD publishes a pinned install manifest per release; use the current stable tag if v2.13 has moved on. The rest of the guide uses features that have existed since 2.x.
  3. Expose the Argo CD UI through an Ingress (insecure mode behind the ingress, which is fine for a local lab) and log in with the CLI.
    bash
    kubectl -n argocd patch configmap argocd-cmd-params-cm --type merge -p '{"data":{"server.insecure":"true"}}'
kubectl -n argocd rollout restart deploy/argocd-server
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd
  namespace: argocd
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  ingressClassName: nginx
  rules:
    - host: argocd.localtest.me
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 80
EOF
brew install argocd    # or the release binary on Linux/Windows
PASS=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)
argocd login argocd.localtest.me --username admin --password "$PASS" --plaintext
argocd account update-password --current-password "$PASS" --new-password 'pick-a-long-one'
    Check: http://argocd.localtest.me opens the UI (localtest.me resolves to 127.0.0.1); argocd app list returns an empty list. localtest.me and its subdomains resolve to localhost without any hosts-file edits.
Phase 2

A Helm chart for the app

The zero-to-production API packaged as a chart with sensible defaults, probes, resources, a ServiceMonitor and per-environment values, tested locally with Helm before Argo CD ever sees it.

  1. In the app repository, scaffold a chart and replace the generated templates with a minimal, explicit set.
    bash
    cd zero-to-prod && git switch -c feat/helm-chart
helm create chart && rm -rf chart/templates/* chart/charts chart/values.yaml
  2. Write Chart.yaml and values.yaml. The image is referenced by digest, and the values file is what the config repo overrides per environment.
    yaml
    # chart/Chart.yaml
apiVersion: v2
name: orders-api
description: The zero-to-production FastAPI service
type: application
version: 0.1.0
appVersion: "0.1.0"

# chart/values.yaml
image:
  repository: ghcr.io/YOUR_GITHUB_USER/zero-to-prod
  digest: ""            # set per environment by the config repo; empty means :latest for local testing
  tag: latest
replicaCount: 2
service:
  port: 80
  targetPort: 8000
ingress:
  enabled: true
  className: nginx
  host: orders.localtest.me
resources:
  requests: {cpu: 100m, memory: 128Mi}
  limits: {memory: 256Mi}
env:
  GREETING: "hello from kubernetes"
metrics:
  serviceMonitor: true
autoscaling:
  enabled: false
  minReplicas: 2
  maxReplicas: 6
  targetCPU: 70
  3. Write the Deployment template with probes, the restricted security context and the image reference logic.
    helm
    # chart/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
  labels: {app.kubernetes.io/name: {{ .Chart.Name }}, app.kubernetes.io/instance: {{ .Release.Name }}}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels: {app.kubernetes.io/instance: {{ .Release.Name }}}
  template:
    metadata:
      labels: {app.kubernetes.io/name: {{ .Chart.Name }}, app.kubernetes.io/instance: {{ .Release.Name }}}
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: app
          image: "{{ .Values.image.repository }}{{ if .Values.image.digest }}@{{ .Values.image.digest }}{{ else }}:{{ .Values.image.tag }}{{ end }}"
          ports: [{name: http, containerPort: 8000}]
          env:
            {{- range $k, $v := .Values.env }}
            - name: {{ $k }}
              value: {{ $v | quote }}
            {{- end }}
          readinessProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 5
          livenessProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 15
          resources: {{- toYaml .Values.resources | nindent 12 }}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: [ALL]}
    Add service.yaml, ingress.yaml (host from values, path /), hpa.yaml (guarded by .Values.autoscaling.enabled) and servicemonitor.yaml (guarded by .Values.metrics.serviceMonitor, selecting the Service, endpoint http, path /metrics). The ServiceMonitor's API group is monitoring.coreos.com/v1, which exists once the monitoring stack is installed in Phase 4; keep it disabled in dev values until then.
  4. Add the remaining templates: the Service, the Ingress and the ServiceMonitor. Each is small; the guard on the ServiceMonitor lets the chart install before the monitoring CRDs exist.
    helm
    # chart/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}
  labels: {app.kubernetes.io/instance: {{ .Release.Name }}}
spec:
  selector: {app.kubernetes.io/instance: {{ .Release.Name }}}
  ports:
    - name: http
      port: {{ .Values.service.port }}
      targetPort: http
---
# chart/templates/ingress.yaml
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}
spec:
  ingressClassName: {{ .Values.ingress.className }}
  rules:
    - host: {{ .Values.ingress.host }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ .Release.Name }}
                port: {name: http}
{{- end }}
---
# chart/templates/servicemonitor.yaml
{{- if .Values.metrics.serviceMonitor }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ .Release.Name }}
  labels: {release: monitoring}
spec:
  selector:
    matchLabels: {app.kubernetes.io/instance: {{ .Release.Name }}}
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
{{- end }}
    Check: helm template orders-api chart renders four documents (Deployment, Service, Ingress, ServiceMonitor) with no template errors.
  5. Lint and render the chart, then install it into the cluster by hand once to prove it works — the last manual install you will do.
    bash
    helm lint chart
helm template orders-api chart --set metrics.serviceMonitor=false | kubectl apply --dry-run=server -f -
helm upgrade --install orders-api chart -n orders --create-namespace --set metrics.serviceMonitor=false
kubectl -n orders rollout status deploy/orders-api
curl -s http://orders.localtest.me/health
    Check: {"status":"ok"}. Uninstall afterwards (helm uninstall orders-api -n orders) so Argo CD owns it from here.
  6. Add a chart test job in CI (helm lint and a server-side dry run against a kind cluster in the workflow) and merge the chart through a pull request.
    bash
    git add chart && git commit -m "feat: Helm chart for orders-api" && git push -u origin feat/helm-chart
gh pr create --fill && gh pr merge --squash --delete-branch && git switch main && git pull
    For the CI job use helm/kind-action@v1 to spin up a cluster, then the same two commands as above.
Phase 3

The config repository and Argo CD Applications

A separate repository describing dev and prod as Kustomize overlays of the chart, and Argo CD Applications that sync them, created through an app-of-apps so even the Applications are in git.

  1. Create the config repository. Separating configuration from application code is the GitOps convention: the app repo changes when code changes, the config repo changes when what runs changes, and the second is what Argo CD watches.
    bash
    cd .. && gh repo create zero-to-prod-config --public --clone && cd zero-to-prod-config
mkdir -p apps/orders-api/{base,dev,prod} platform/{monitoring,ingress-nginx} argocd/applications
  2. Write the base as a Kustomize HelmChart inflation pointing at the chart in the app repository, and the environment overlays that pin the image digest and set environment-specific values.
    yaml
    # apps/orders-api/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
  - name: orders-api
    repo: oci://ghcr.io/YOUR_GITHUB_USER/charts      # publish the chart as an OCI artifact from CI (Phase 5)
    version: 0.1.0
    releaseName: orders-api
    namespace: orders
    valuesFile: values.yaml

# apps/orders-api/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: orders-dev
resources: [../base]
patches:
  - target: {kind: Deployment, name: orders-api}
    patch: |
      - op: replace
        path: /spec/replicas
        value: 1
images:
  - name: ghcr.io/YOUR_GITHUB_USER/zero-to-prod
    digest: sha256:REPLACE_WITH_A_REAL_DIGEST

# apps/orders-api/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: orders-prod
resources: [../base]
images:
  - name: ghcr.io/YOUR_GITHUB_USER/zero-to-prod
    digest: sha256:REPLACE_WITH_A_REAL_DIGEST
    Until the chart is published to an OCI registry in Phase 5, use a local path instead: copy chart/ into apps/orders-api/base/chart and reference it with helmCharts[0].repo omitted and name: chart. Get a real digest with docker buildx imagetools inspect ghcr.io/YOUR_GITHUB_USER/zero-to-prod:latest. Set the ingress host per environment in each overlay's values (orders-dev.localtest.me, orders.localtest.me).
  3. Define the Argo CD project (what may be deployed where) and the Applications for both environments. Dev syncs automatically with pruning and self-heal; prod syncs automatically too but only after a change lands in its own directory, which the release flow gates.
    yaml
    # argocd/project.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: zero-to-prod
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/YOUR_GITHUB_USER/zero-to-prod-config
    - https://prometheus-community.github.io/helm-charts
    - oci://ghcr.io/YOUR_GITHUB_USER/charts
  destinations:
    - {namespace: "orders-*", server: https://kubernetes.default.svc}
    - {namespace: monitoring, server: https://kubernetes.default.svc}
  clusterResourceWhitelist:
    - {group: "", kind: Namespace}
    - {group: apiextensions.k8s.io, kind: CustomResourceDefinition}
    - {group: rbac.authorization.k8s.io, kind: ClusterRole}
    - {group: rbac.authorization.k8s.io, kind: ClusterRoleBinding}

# argocd/applications/orders-api-dev.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-api-dev
  namespace: argocd
spec:
  project: zero-to-prod
  source:
    repoURL: https://github.com/YOUR_GITHUB_USER/zero-to-prod-config
    targetRevision: main
    path: apps/orders-api/dev
  destination: {server: https://kubernetes.default.svc, namespace: orders-dev}
  syncPolicy:
    automated: {prune: true, selfHeal: true}
    syncOptions: [CreateNamespace=true]

# argocd/applications/orders-api-prod.yaml: same with name orders-api-prod, path apps/orders-api/prod,
# namespace orders-prod, and syncPolicy.automated.selfHeal: true, prune: false (prod deletions are manual)
    Argo CD needs Kustomize's Helm support enabled: patch argocd-cm with kustomize.buildOptions: --enable-helm, then restart the repo server.
  4. Write the app-of-apps root Application, apply only that one manually, and watch Argo CD create the rest from the repository.
    yaml
    # argocd/app-of-apps.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR_GITHUB_USER/zero-to-prod-config
    targetRevision: main
    path: argocd
    directory: {recurse: true, exclude: app-of-apps.yaml}
  destination: {server: https://kubernetes.default.svc, namespace: argocd}
  syncPolicy:
    automated: {prune: true, selfHeal: true}
  5. Before committing, render each overlay locally exactly as Argo CD will, so a Kustomize or Helm mistake is caught on your laptop rather than as a red Application.
    bash
    kustomize build --enable-helm apps/orders-api/dev | kubectl apply --dry-run=server -f -
kustomize build --enable-helm apps/orders-api/prod | grep -A1 'image:'
    Check: Both overlays render; the prod output shows the image with the @sha256: digest you pinned.
  6. Commit the config repository, enable Kustomize's Helm support, and bootstrap.
    bash
    git add . && git commit -m "feat: orders-api dev/prod overlays, AppProject, Applications, app-of-apps" && git push -u origin main
kubectl -n argocd patch configmap argocd-cm --type merge -p '{"data":{"kustomize.buildOptions":"--enable-helm"}}'
kubectl -n argocd rollout restart deploy/argocd-repo-server
kubectl apply -f argocd/app-of-apps.yaml
argocd app list
argocd app wait orders-api-dev --health --timeout 300
    Check: argocd app list shows root, orders-api-dev and orders-api-prod as Synced and Healthy; curl http://orders-dev.localtest.me/health works; the UI shows the tree of Deployment, ReplicaSet, Pods, Service and Ingress under each app.
Phase 4

The monitoring stack, the same way

Prometheus, Alertmanager and Grafana installed and upgraded by Argo CD from a chart reference in git, scraping the app through its ServiceMonitor.

  1. Add a multi-source Application for kube-prometheus-stack: the chart from the community repository, the values from your config repository.
    yaml
    # argocd/applications/monitoring.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: monitoring
  namespace: argocd
spec:
  project: zero-to-prod
  sources:
    - repoURL: https://prometheus-community.github.io/helm-charts
      chart: kube-prometheus-stack
      targetRevision: 67.9.0
      helm:
        releaseName: monitoring
        valueFiles: [$values/platform/monitoring/values.yaml]
    - repoURL: https://github.com/YOUR_GITHUB_USER/zero-to-prod-config
      targetRevision: main
      ref: values
  destination: {server: https://kubernetes.default.svc, namespace: monitoring}
  syncPolicy:
    automated: {prune: true, selfHeal: true}
    syncOptions: [CreateNamespace=true, ServerSideApply=true]

# platform/monitoring/values.yaml
grafana:
  adminPassword: change-me-in-a-secret      # lab only; use a SecretRef or an external secrets operator for real
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts: [grafana.localtest.me]
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false     # scrape ServiceMonitors from every namespace
    retention: 7d
alertmanager:
  enabled: true
    ServerSideApply=true avoids the annotation-size limit that the stack's large CRDs otherwise hit. Pin the chart version and bump it via pull requests like any dependency.
  2. Commit, let the root app pick it up, and wait for the stack to become healthy. Then enable the ServiceMonitor in both overlays' values and confirm Prometheus scrapes the app.
    bash
    git add . && git commit -m "feat: kube-prometheus-stack via Argo CD" && git push
argocd app wait monitoring --health --timeout 600
# now set metrics.serviceMonitor: true in apps/orders-api/{dev,prod} values and push
argocd app wait orders-api-dev --sync
    Check: http://grafana.localtest.me logs in; Prometheus (port-forward kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090) shows orders-dev/orders-api and orders-prod/orders-api targets as UP.
  3. Add the SLO dashboard from the SRE project as a ConfigMap with the grafana_dashboard: "1" label in platform/monitoring/, and the burn-rate rules as a PrometheusRule resource. Both are just files in git now; Argo CD applies them and the operator loads them.
    yaml
    # platform/monitoring/orders-slo-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: orders-api-slo
  namespace: monitoring
  labels: {release: monitoring}
spec:
  groups:
    - name: orders_slo
      rules:
        - record: sli:availability_error_ratio:rate5m
          expr: |
            sum(rate(http_requests_total{namespace="orders-prod", status=~"5.."}[5m]))
            / sum(rate(http_requests_total{namespace="orders-prod"}[5m]))
        - alert: OrdersApiAvailabilityFastBurn
          expr: sli:availability_error_ratio:rate5m > 0.0144
          for: 2m
          labels: {severity: page}
          annotations: {summary: "orders-api (prod) burning availability budget fast"}
    Reference the platform directory from an Application (platform-monitoring-extras, path platform/monitoring, excluding values.yaml) so these manifests sync. The full rule set from the SLO project drops in unchanged apart from the namespace selector.
Phase 5

The release flow, drift and rollback

CI publishes the chart and the image, then opens a pull request in the config repo that bumps the dev digest; a second pull request promotes to prod; and you see Argo CD revert drift and roll back a bad release.

  1. Protect the config repository's main branch the same way the app repo is protected: pull requests only, and a required review for anything touching apps/orders-api/prod/. A CODEOWNERS file makes the prod directory require a named reviewer.
    bash
    printf 'apps/orders-api/prod/ @YOUR_GITHUB_USER\nargocd/ @YOUR_GITHUB_USER\n' > CODEOWNERS
git add CODEOWNERS && git commit -m "chore: code owners for prod and argocd" && git push
gh api -X PUT "repos/{owner}/zero-to-prod-config/branches/main/protection" --input - <<'JSON'
{"required_status_checks": null, "enforce_admins": false,
 "required_pull_request_reviews": {"require_code_owner_reviews": true, "required_approving_review_count": 1},
 "restrictions": null, "allow_force_pushes": false, "allow_deletions": false}
JSON
    Working alone, you can approve your own bot's dev bumps but GitHub will not let you approve your own prod pull requests; that is the point of the rule. For a solo lab, drop required_approving_review_count to 0 for dev-only changes if it gets in the way.
  2. In the app repository's release.yml, after the image is pushed: package and push the chart to GHCR as an OCI artifact, then update the dev overlay's digest in the config repo through a pull request. A fine-grained token for the config repo lives in the app repo's secrets as CONFIG_REPO_TOKEN.
    yaml
      promote-dev:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Publish the chart as an OCI artifact
        run: |
          echo "${{ secrets.GITHUB_TOKEN }}" | helm registry login ghcr.io -u "${{ github.actor }}" --password-stdin
          helm package chart --version "0.1.${{ github.run_number }}" --app-version "${{ needs.build.outputs.tag }}"
          helm push orders-api-0.1.${{ github.run_number }}.tgz oci://ghcr.io/${{ github.repository_owner }}/charts
      - name: Bump the dev digest in the config repo
        env:
          GH_TOKEN: ${{ secrets.CONFIG_REPO_TOKEN }}
          DIGEST: ${{ needs.build.outputs.digest }}
        run: |
          gh repo clone ${{ github.repository_owner }}/zero-to-prod-config cfg
          cd cfg && git switch -c "bump/dev-${DIGEST:7:12}"
          sed -i "s#digest: sha256:.*#digest: ${DIGEST}#" apps/orders-api/dev/kustomization.yaml
          sed -i "s#version: 0.1.*#version: 0.1.${{ github.run_number }}#" apps/orders-api/base/kustomization.yaml
          git -c user.name=release-bot -c user.email=bot@users.noreply.github.com commit -am "dev: ${{ github.sha }} (${DIGEST:7:12})"
          git push -u origin HEAD
          gh pr create --fill --body "Image ${DIGEST} from ${{ github.server_url }}/${{ github.repository }}/commit/${{ github.sha }}"
    docker/build-push-action exposes outputs.digest; add it to the build job's outputs. The bot opens a pull request rather than pushing to main so that the config repo's branch protection applies to machines too. For dev you may auto-merge it; for prod a human merges.
  3. Make a small code change in the app, merge it, and follow the chain: the image is built, the chart is pushed, a bump pull request appears in the config repo. Merge it and watch Argo CD sync dev.
    bash
    cd ../zero-to-prod && git switch -c feat/greeting-k8s
sed -i.bak 's/hello from zero-to-prod/hello from gitops/' app/main.py && rm app/main.py.bak
git commit -am "feat: gitops greeting" && git push -u origin HEAD && gh pr create --fill && gh pr merge --squash --delete-branch
gh run watch
cd ../zero-to-prod-config && gh pr list && gh pr merge --squash --delete-branch $(gh pr list --json number --jq '.[0].number')
argocd app wait orders-api-dev --sync --health
curl -s http://orders-dev.localtest.me/
    Check: The response changes in dev only. Prod still serves the old digest — the point of separate overlays.
  4. Promote to prod as a git operation: copy the digest from dev to prod in a pull request that a human merges.
    bash
    git switch -c promote/prod
DIG=$(grep -o 'sha256:[0-9a-f]*' apps/orders-api/dev/kustomization.yaml)
sed -i "s#digest: sha256:.*#digest: ${DIG}#" apps/orders-api/prod/kustomization.yaml
git commit -am "prod: promote ${DIG:7:12}" && git push -u origin HEAD && gh pr create --fill
# a reviewer merges; then:
argocd app wait orders-api-prod --sync --health && curl -s http://orders.localtest.me/
    Check: Prod now serves the new greeting. The config repo's history is the deployment log: who promoted what, when, and from which app commit.
  5. Drift: change the cluster by hand and watch Argo CD put it back.
    bash
    kubectl -n orders-prod scale deploy/orders-api --replicas=5
sleep 20 && kubectl -n orders-prod get deploy orders-api
argocd app history orders-api-prod | tail -3
    Check: Replicas return to the value in git within a few seconds (self-heal); the UI briefly showed OutOfSync. Manual changes to production are now impossible to keep, which is the feature.
  6. Rollback: ship a bad release (make /health return 500 in a branch, merge, promote), watch prod's health degrade in Argo CD, then revert the promotion commit.
    bash
    # after promoting the bad digest:
argocd app get orders-api-prod | grep -E 'Health|Sync'          # Degraded: readiness fails, rollout stuck
git revert --no-edit HEAD && git push
argocd app wait orders-api-prod --sync --health
curl -s http://orders.localtest.me/health
    Check: Health returns to Healthy and the previous digest is running. argocd app rollback exists too, but the git revert is the GitOps answer because the repository stays the truth.
  7. Write the README for the config repo: the promotion flow, who may merge prod, how to roll back, how to add a new app (a directory plus an Application file), and the one rule — nobody runs kubectl apply against this cluster. Then delete the kind cluster when you are done.
    bash
    kind delete cluster --name gitops
    Rebuilding is kind create cluster, the ingress install, the Argo CD install, and kubectl apply -f argocd/app-of-apps.yaml: everything else comes back from git. That is the recoverability test.
Help

Troubleshooting

Argo CD shows rpc error: ... helm chart inflation or ignores helmCharts
Kustomize's Helm support is off by default in Argo CD; patch argocd-cm with kustomize.buildOptions: --enable-helm and restart argocd-repo-server.
The kube-prometheus-stack Application fails with metadata.annotations: Too long
Enable ServerSideApply=true in the Application's sync options (the CRDs exceed the client-side apply annotation limit).
Prometheus does not scrape the app although the ServiceMonitor exists
Set serviceMonitorSelectorNilUsesHelmValues: false in the stack's values, or label the ServiceMonitor with release: monitoring. Check kubectl -n monitoring get servicemonitors -A and the Prometheus UI's Service Discovery page.
orders.localtest.me does not resolve
Some corporate DNS setups block wildcard public DNS to localhost. Add 127.0.0.1 orders.localtest.me argocd.localtest.me grafana.localtest.me to /etc/hosts instead.
The bump pull request is not created
The CONFIG_REPO_TOKEN must be a fine-grained token with Contents and Pull requests write on the config repo; gh also needs GH_TOKEN set in the step's env, as shown.
Argo CD reports OutOfSync forever on a resource it did not create
Mutating webhooks and defaulted fields cause diffs; add an ignoreDifferences entry for the field, or RespectIgnoreDifferences=true in sync options. Check the diff in the UI to see what changes.
The prod Application pruned something it should not have
Prod was configured with prune: false for this reason; if you enabled it, a resource removed from git is deleted from the cluster. Restore the file in git; that is the recovery path.
Next

Where to go from here

Did a step fail or feel unclear? Tell me which one →