Infrastructure as Code with Terraform

Declarative infrastructure

Terraform configuration, written in HCL, declares the infrastructure you want. Terraform compares that with what it believes exists, works out the difference, and makes the API calls to close it. This is the same desired-state idea as Kubernetes, with one important difference: Terraform reconciles only when you run it, not continuously.

A provider is a plugin that translates HCL into calls to one API: AWS, Azure, Google Cloud, Cloudflare, GitHub, Kubernetes and thousands more. A resource is one thing the provider manages. Terraform builds a dependency graph from the references between resources, so it creates things in the right order and in parallel where it can.

terraform {
  required_version = ">= 1.6"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.region
}

resource "aws_s3_bucket" "artifacts" {
  bucket = "acme-${var.environment}-artifacts"
  tags   = local.common_tags
}

resource "aws_s3_bucket_versioning" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id      # a reference: creates the dependency
  versioning_configuration {
    status = "Enabled"
  }
}

A resource address has the form TYPE.NAME, here aws_s3_bucket.artifacts. The name is local to your configuration and is how other resources refer to it. ~> 5.0 allows any 5.x provider release but not 6.0, and terraform init records the exact version chosen in .terraform.lock.hcl, which you commit, for the same reproducibility reasons as any other lockfile.

The core workflow

terraform init              # download providers and modules, configure the backend
terraform fmt -recursive     # canonical formatting
terraform validate           # syntax and internal consistency
terraform plan -out=tfplan   # compute the changes and save them
terraform apply tfplan       # execute exactly the saved plan
terraform destroy            # remove everything this configuration manages

The plan is the heart of Terraform's safety. It lists every change before anything happens, using symbols you must learn to read.

Some attributes cannot be changed on a live resource, so changing them makes Terraform replace it. Renaming a database instance can therefore mean deleting the database. Read the summary line, Plan: 2 to add, 1 to change, 0 to destroy, every single time, and treat any unexpected destroy as a stop sign.

State: Terraform's memory

Terraform records every resource it manages in a state file, mapping addresses such as aws_s3_bucket.artifacts to real IDs in the provider. Without state it could not know that a bucket already exists, which resources to delete when you remove a block, or what depends on what. By default state is a local terraform.tfstate file, which fails the moment a second person or a pipeline is involved.

Teams use a remote backend: shared storage for the state, with locking so two applies cannot run at once and corrupt it.

terraform {
  backend "s3" {
    bucket       = "acme-terraform-state"
    key          = "shop/prod/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true
  }
}

• State contains secrets. Generated passwords, private keys and connection strings are stored in it in plain text. Encrypt the backend, restrict who can read it, and never commit state to Git.
• Never edit state by hand. Use the terraform state subcommands, or better, the declarative moved, import and removed blocks.
• Turn on versioning for the storage that holds state, so a bad write can be recovered.
• Keep states small. One state for the whole company means every plan is slow and every mistake has an enormous blast radius. Split by environment and by layer, for example network, cluster and application.

Variables, locals, outputs and data sources

variable "environment" {
  type        = string
  description = "Deployment environment"
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging or prod."
  }
}

variable "instance_count" {
  type    = number
  default = 2
}

variable "db_password" {
  type      = string
  sensitive = true          # redacted in plan output; still stored in state
}

locals {
  common_tags = {
    Environment = var.environment
    ManagedBy   = "terraform"
    Team        = "shop"
  }
}

data "aws_caller_identity" "current" {}     # read-only lookup of something that exists

output "bucket_name" {
  value = aws_s3_bucket.artifacts.bucket
}

Variables are a configuration's inputs, outputs are its results, and locals are named expressions that save you repeating yourself. A data source reads information about something Terraform does not manage, such as the current account, the newest machine image or an existing network.

Variable values are resolved in a fixed order, with later sources winning: the default, then TF_VAR_name environment variables, then terraform.tfvars, then *.auto.tfvars, then -var-file and -var on the command line. Pipelines typically pass secrets as TF_VAR_ environment variables and everything else in a per-environment .tfvars file.

# one resource per item, addressed by key: aws_iam_user.team["alice"]
resource "aws_iam_user" "team" {
  for_each = toset(["alice", "bob", "carol"])
  name     = each.key
}

Prefer for_each to count for collections. With count, resources are addressed by position, so removing the first item shifts every index and Terraform plans to destroy and recreate the rest. With for_each they are addressed by key, and removing one touches only that one.

Modules and environments

A module is a directory of .tf files with variables as inputs and outputs as results, the unit of reuse. The directory where you run Terraform is the root module; it calls child modules. Write a module when you need the same group of resources more than once, or want to enforce a standard such as "every bucket is encrypted and versioned".

module "network" {
  source = "./modules/network"

  environment = var.environment
  cidr_block  = "10.20.0.0/16"
}

module "cluster" {
  source  = "terraform-aws-modules/eks/aws"   # from the public registry
  version = "~> 20.0"                         # always pin registry modules

  cluster_name = "shop-${var.environment}"
  vpc_id       = module.network.vpc_id         # a module output
  subnet_ids   = module.network.private_subnet_ids
}

For environments, the robust pattern is one directory per environment, each with its own backend and therefore its own state, all calling the same modules with different inputs.

infra/
  modules/
    network/   cluster/   database/
  envs/
    dev/       main.tf  backend.tf  terraform.tfvars
    staging/   main.tf  backend.tf  terraform.tfvars
    prod/      main.tf  backend.tf  terraform.tfvars

Separate states mean that a mistake in dev cannot touch prod, and you can grant different people access to each. Terraform workspaces offer several states for one configuration, but every workspace shares the same backend and code, which makes it easy to apply to the wrong one. They suit short-lived copies better than the split between production and everything else.

Drift, imports and guard rails

Drift is when real infrastructure no longer matches the configuration because someone changed it by hand. terraform plan reveals it as an unexpected change. You then either revert the manual change by applying, or accept it by updating the code. Running a scheduled plan in CI and alerting on any difference catches drift early.

# adopt an existing resource into state without recreating it
import {
  to = aws_s3_bucket.legacy
  id = "acme-legacy-uploads"
}

# rename or move a resource in code without destroying it
moved {
  from = aws_s3_bucket.artifacts
  to   = aws_s3_bucket.build_artifacts
}

resource "aws_db_instance" "main" {
  # ...arguments...
  lifecycle {
    prevent_destroy = true        # any plan that would destroy this fails
    ignore_changes  = [tags["LastPatched"]]
  }
}

Renaming a resource block without a moved block looks to Terraform like "delete the old one, create a new one", which for a database is catastrophic. Put prevent_destroy on anything stateful.

In a team, nobody applies from a laptop. A pull request runs fmt -check, validate, a linter such as tflint, a security scanner, and plan, posting the plan for review. After merge, the pipeline applies that plan using short-lived cloud credentials obtained through OIDC. OpenTofu, the open-source fork of Terraform, uses the same language and workflow, so everything here applies to it too.