GitOps with Argo CD

The GitOps model

GitOps is an operating model, not a product. The OpenGitOps project summarises it in four principles.

1. Declarative. The desired state of the system is expressed declaratively, as Kubernetes manifests, Helm charts or Kustomize overlays.
2. Versioned and immutable. That desired state is stored so that history is kept and versions cannot be altered. In practice, Git.
3. Pulled automatically. Software agents pull the desired state from the source; nothing external pushes into the cluster.
4. Continuously reconciled. Agents constantly compare actual with desired state and act to converge them.

The security gain is significant. A CI system that can deploy to production is a high-value target, because compromising it compromises the cluster. With GitOps, CI needs permission only to push to a Git repository and an image registry. The cluster reaches out; nothing reaches in.

You already know the underlying idea. It is the Kubernetes reconciliation loop, extended one level up, so that Git, and no longer whoever last ran kubectl apply, defines what the cluster should be. Argo CD and Flux are the two main implementations, and both are CNCF graduated projects.

Installing Argo CD and defining an Application

kubectl create namespace argocd
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

kubectl -n argocd get pods
kubectl -n argocd port-forward svc/argocd-server 8080:443      # UI on https://localhost:8080

# initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d

The central object is the Application, a custom resource that links a source, which is a repository, a revision and a path, to a destination, which is a cluster and a namespace. Because it is itself a Kubernetes manifest, it belongs in Git too.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-staging
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/shop-deploy.git
    targetRevision: main
    path: apps/orders/overlays/staging
  destination:
    server: https://kubernetes.default.svc
    namespace: shop-staging
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Argo CD detects what the path contains. Plain YAML is applied as it is, a kustomization.yaml is built with Kustomize, and a Chart.yaml is rendered with Helm. For Helm, note that Argo CD runs the equivalent of helm template and applies the output. It does not create Helm releases, so helm list will not show them.

source:
  repoURL: https://github.com/acme/shop-deploy.git
  targetRevision: main
  path: charts/orders
  helm:
    valueFiles:
      - values.yaml
      - values-prod.yaml

Sync status, health and the three switches

Argo CD reports two independent things about every Application, and mixing them up causes a lot of confusion.

Synced but Degraded means Argo CD applied exactly what you committed and what you committed is broken, for example an image that crash-loops. OutOfSync but Healthy means the cluster works but differs from Git: either a commit has not been applied yet, or someone changed the cluster by hand.

Three settings under syncPolicy.automated decide how assertive Argo CD is.

• automated present: new commits are applied without anyone pressing Sync. Without it, Argo CD only reports OutOfSync and waits.
• prune: true: a resource deleted from Git is deleted from the cluster. It is off by default, because deleting is dangerous, but without it removed manifests linger forever.
• selfHeal: true: manual changes to managed resources are reverted to match Git. This is what makes Git the real source of truth instead of a suggestion.

argocd login localhost:8080
argocd app list
argocd app get orders-staging              # sync and health, resource by resource
argocd app diff orders-staging             # what differs between Git and the cluster
argocd app sync orders-staging             # apply now
argocd app history orders-staging          # deployed revisions
argocd app wait orders-staging --health    # block until healthy (useful in scripts)

Repositories and promotion

Keep deployment configuration in a separate repository from application source code. The two change at different rates and for different reasons, and keeping them apart stops every image bump from triggering the application's CI, which would otherwise loop. It also lets you give fewer people write access to what runs in production.

shop-deploy/
  apps/
    orders/
      base/                     # Deployment, Service, shared by all environments
        kustomization.yaml
      overlays/
        staging/kustomization.yaml    # image tag + staging patches
        prod/kustomization.yaml       # image tag + prod patches
  argocd/
    orders-staging.yaml         # the Application objects themselves
    orders-prod.yaml

Use directories per environment, not long-lived branches. Branch-per-environment looks tidy but leads to merge conflicts, drift between branches and no clear answer to what differs between staging and production. With directories, that answer is a diff of two folders.

The full delivery flow then looks like this.

1. A developer merges to the application repository. CI tests, builds the image, tags it with the commit SHA and pushes it.
2. CI's last step commits one line to the deploy repository: the new tag in overlays/staging. That commit is the deployment.
3. Argo CD sees the commit and syncs staging. Health checks and smoke tests run.
4. Promotion is a pull request that copies the same tag into overlays/prod. Review and merge are the approval gate, and the merge is the production deploy.

git clone https://github.com/acme/shop-deploy.git && cd shop-deploy/apps/orders/overlays/staging
kustomize edit set image ghcr.io/acme/orders-api="ghcr.io/acme/orders-api:${GIT_SHA}"
git commit -am "orders: deploy ${GIT_SHA} to staging"
git push

When you have dozens of Applications, do not create them by hand. The app-of-apps pattern uses one parent Application whose source is a directory of Application manifests, and an ApplicationSet generates Applications from a template, for example one per directory in the repository or one per cluster.

Rollbacks, secrets and drift

Rollback is git revert. Reverting the commit that introduced the bad version makes Git describe the previous state again, and Argo CD applies it. The rollback is reviewed and audited like any other change, which is the same rule as in the Git module: shared history moves forward. Argo CD also has argocd app rollback, but it requires automated sync to be off and leaves Git out of step with the cluster, so treat it as an emergency measure.

Secrets are the hard part, because everything must be in Git and secrets must not be readable there. There are two established answers.

• Encrypt what you commit. Sealed Secrets encrypts a Secret with a key that only the in-cluster controller holds, so the sealed form is safe in Git. SOPS encrypts the values of a YAML file with a cloud KMS key or an age key.
• Commit only a reference. External Secrets Operator reads a manifest that says which entry in AWS Secrets Manager, Vault or a similar store to fetch, and creates the Kubernetes Secret inside the cluster. Nothing sensitive ever touches Git, and rotation happens in one place.

Drift becomes visible and manageable. Some differences are legitimate: a HorizontalPodAutoscaler changes spec.replicas, and a controller may inject fields. If you declare a replica count in Git and an autoscaler manages it, the two fight, and the Application flaps between Synced and OutOfSync. Omit replicas from the manifest, or tell Argo CD to ignore that field.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas