CI/CD pipelines

CI, CD and the other CD

Continuous integration (CI) means every developer merges to the main branch at least daily, and every push is automatically built and tested. Its purpose is fast feedback: you learn that you broke something within minutes, while the change is still in your head.

Continuous delivery means every commit that passes the pipeline is a release candidate that could go to production at the push of a button. Continuous deployment removes the button: every passing commit goes live automatically. Both share the abbreviation CD; the difference is whether a human makes the final decision.

• Keep it fast. If the pipeline takes forty minutes, people batch changes and stop waiting for it. Aim for feedback on a pull request in about ten minutes.
• Fail early. Run the cheapest checks first: lint and unit tests before integration tests before anything that deploys.
• Keep main green. A red main branch blocks everyone. Fixing it, or reverting, takes priority over new work.
• Build once. The artifact that was tested is the artifact that ships. Rebuilding per environment means you deploy something nobody tested.

Anatomy of a workflow

GitHub Actions reads YAML files in .github/workflows/. A workflow is triggered by an event (on:). It contains jobs, which run in parallel by default, each on a fresh virtual machine called a runner. A job is a list of steps, run in order on that one machine. A step either runs shell commands (run:) or calls a reusable action (uses:).

name: CI

on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: ruff check .
      - run: pytest --maxfail=1

A step fails when its command exits non-zero, which is the exit-code convention from the scripting module doing its job. A failed step fails the job, and later steps are skipped. Because each job gets a brand-new machine, nothing carries over between jobs unless you pass it explicitly as an artifact, a cache entry or a job output.

Faster: caching, matrices, dependencies between jobs

Downloading the same dependencies on every run is the most common waste. The setup-* actions have a built-in cache: option, and the general actions/cache action stores any directory under a key. Derive the key from a hash of your lockfile, so the cache is reused until your dependencies actually change.

A matrix runs the same job once for each combination of values, in parallel: several language versions, several operating systems. needs: makes one job wait for another, which turns the default parallel jobs into an ordered pipeline.

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
        node: [20, 22]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
          cache: npm
      - run: npm ci
      - run: npm test

  build:
    needs: test              # runs only if every matrix leg passed
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/

npm ci installs exactly what the lockfile says and fails if it is out of date, which is what you want in CI. The concurrency key cancels superseded runs so that pushing three commits in a row does not queue three full pipelines.

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

Secrets, permissions and untrusted code

Credentials are stored as encrypted secrets at repository, organisation or environment level and read with ${{ secrets.NAME }}. They are masked in logs. Non-sensitive settings go in variables (${{ vars.NAME }}). Every run also receives an automatic, short-lived GITHUB_TOKEN; restrict what it can do with a permissions: block and grant only what each job needs.

permissions:
  contents: read            # default for every job in this workflow

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write       # only this job may push images
    steps:
      - uses: actions/checkout@v4
      - name: Log in to the container registry
        run: echo "$TOKEN" | docker login ghcr.io -u "$ACTOR" --password-stdin
        env:
          TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ACTOR: ${{ github.actor }}

• Workflows triggered by pull_request from a fork do not receive your secrets. That is deliberate: a stranger's code must not be able to read them.
• A third-party action is code you run with access to your token. Prefer well-known publishers, and for sensitive pipelines pin to a full commit SHA instead of a movable tag.
• Prefer short-lived cloud credentials obtained through OIDC over long-lived access keys stored as secrets. The secrets module covers this.

Build once, promote everywhere

The delivery half of the pipeline packages the application as an immutable artifact, usually a container image, tags it with something traceable such as the commit SHA, and then moves that same image through staging and production. Differences between environments live in configuration, never in the build.

An environment in GitHub Actions is a named deployment target with its own secrets and protection rules. Add required reviewers to production and the job pauses until someone approves it, which is continuous delivery's "push of a button".

name: Deliver
on:
  push:
    branches: [main]

env:
  IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - name: Log in to the registry
        run: echo "$TOKEN" | docker login ghcr.io -u "$ACTOR" --password-stdin
        env:
          TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ACTOR: ${{ github.actor }}
      - run: docker build -t "$IMAGE" .
      - run: docker push "$IMAGE"

  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/deploy.sh staging "$IMAGE"
      - run: ./scripts/smoke-test.sh https://staging.example.com

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production      # waits for required reviewers
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/deploy.sh production "$IMAGE"

Tagging with the commit SHA gives you traceability in both directions: from a running container you can find the exact source, and rolling back is simply deploying an earlier SHA that already exists in the registry. The smoke test after staging is what earns the right to promote.

When the pipeline itself is the problem

• Flaky tests that pass on a re-run teach the team to ignore red builds. Quarantine them, fix them, or delete them; do not normalise the retry button.
• "Works locally, fails in CI" usually means an undeclared dependency: a tool installed on your laptop, a different runtime version, a file ignored by Git, or a test that depends on timezone or ordering.
• Slow pipelines are fixed by measuring. Look at per-step timings, cache the dependency install, split the test suite across parallel jobs, and move rarely failing slow suites to run after merge.
• Debugging is easier when the logic lives in scripts in your repository (./scripts/test.sh) that the workflow merely calls, because you can run the same script locally. It also keeps you portable across CI systems.

The concepts here are not specific to GitHub. GitLab CI, Jenkins, CircleCI and others all have triggers, jobs, runners, caches, artifacts, secrets and environments under slightly different names.