Data platform infrastructure

The components and their boundaries

A data platform has a small number of moving parts, and most operational pain comes from unclear boundaries between them. Ingestion (connectors, CDC, API pullers) lands raw data. Storage (object storage with a table format, or the warehouse's own storage) holds every layer. Compute runs transformations: the warehouse engine for SQL, Spark or single-node engines for code. Orchestration schedules and records runs. Transformation code (dbt, pipelines) lives in repositories. Governance and observability (catalog, quality, lineage, metrics) sit across all of it. Serving (BI, reverse ETL, feature stores, APIs) reads the outputs. Each is a separate concern with its own identity, its own cost line and its own deploy path.

Infrastructure as code for data

Every resource that outlives a run belongs in Terraform (or the equivalent): buckets with lifecycle rules, warehouse databases, schemas, roles and grants, orchestrator environments, service identities, secrets, network paths, budgets. The reasons are the DevOps track's: reproducibility, review, drift detection, and one place where an auditor can read how access is granted. Warehouse objects are the part teams forget; Terraform providers exist for the major warehouses, and putting roles and grants there means access review is a diff, exactly as the governance module wanted.

resource "aws_s3_bucket" "lake" {
  bucket = "acme-lake-${var.env}"
}

resource "aws_s3_bucket_lifecycle_configuration" "lake" {
  bucket = aws_s3_bucket.lake.id
  rule {
    id     = "raw-tiering"
    status = "Enabled"
    filter { prefix = "raw/" }
    transition { days = 90;  storage_class = "STANDARD_IA" }
    transition { days = 365; storage_class = "GLACIER_IR" }
    expiration { days = 730 }
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "lake" {
  bucket = aws_s3_bucket.lake.id
  rule { apply_server_side_encryption_by_default { sse_algorithm = "aws:kms" } }
}

# warehouse objects via the Snowflake provider: schema, role, and grants reviewed as code
resource "snowflake_schema" "marts" {
  database = var.warehouse_db
  name     = "MARTS"
}

resource "snowflake_account_role" "analyst" { name = "ANALYST" }

resource "snowflake_grant_privileges_to_account_role" "analyst_marts" {
  account_role_name = snowflake_account_role.analyst.name
  privileges        = ["SELECT"]
  on_schema_object {
    future {
      object_type_plural = "TABLES"
      in_schema          = "\"${var.warehouse_db}\".\"${snowflake_schema.marts.name}\""
    }
  }
}

# the identity the orders pipeline runs as: reads raw, writes bronze/silver, nothing else
resource "aws_iam_role" "orders_pipeline" {
  name               = "orders-pipeline-${var.env}"
  assume_role_policy = data.aws_iam_policy_document.orchestrator_assume.json
}

Parameterise by environment (var.env) so the same code produces development, staging and production with different names and sizes. Keep secrets out of state and code: Terraform creates the secret container, a pipeline or a human writes the value, and the workload reads it by identity. Run policy checks on the plan (public buckets, wildcard grants, missing encryption) in CI, as the cloud security module described.

CI/CD for pipelines and models

Pipeline code and dbt projects are software and deploy like it. A pull request runs unit tests on transformation functions (small fixtures, fast), lints SQL and Python, validates DAG files parse, runs the dbt slim build against a CI schema, and runs the governance and contract checks. Merge to main deploys to staging: the orchestrator picks up the new DAGs (from a synced bucket, a container image or a git sync), dbt artifacts are published, and a smoke run executes on a sample. Promotion to production is a tag or a manual approval, and production runs only what is deployed — nobody edits a DAG on the server.

name: data-platform

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.12", cache: pip}
      - run: pip install -r requirements-dev.txt
      - run: ruff check . && sqlfluff lint models/ --dialect snowflake
      - run: pytest tests/unit -q
      - name: DAGs must parse
        run: python -c "from airflow.models import DagBag; b = DagBag('dags', include_examples=False); assert not b.import_errors, b.import_errors"
      - name: dbt slim build
        env:
          DBT_TARGET: ci
        run: |
          dbt deps
          dbt build --select state:modified+ --state ./prod-artifacts --target ci --fail-fast
      - run: python scripts/check_governance_tags.py

  deploy_staging:
    if: github.ref == 'refs/heads/main'
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with: {role-to-assume: "${{ vars.DEPLOY_ROLE }}", aws-region: eu-west-1}
      - run: aws s3 sync dags/ "s3://acme-airflow-staging/dags/" --delete
      - run: dbt docs generate --target staging && aws s3 cp target/manifest.json "s3://acme-dbt-artifacts/staging/manifest.json"

  deploy_prod:
    needs: deploy_staging
    runs-on: ubuntu-latest
    environment: production            # requires an approval in GitHub before this job runs
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with: {role-to-assume: "${{ vars.DEPLOY_ROLE_PROD }}", aws-region: eu-west-1}
      - run: aws s3 sync dags/ "s3://acme-airflow-prod/dags/" --delete

Keep the transformation code, the DAGs, the Terraform and the governance tags in one repository per domain, or a small number of them, so a change to a model and to its schedule and to its grants is one reviewed pull request. Version everything, pin dependencies, and store run artifacts (dbt manifests, test results) after production runs for slim CI and observability.

Environments for data

Software environments are cheap copies; data environments are not, because the data is the expensive part and copying production data everywhere is a privacy problem. The workable pattern: development is a per-developer schema in the warehouse built from a small, masked sample of production (or synthetic data), fast to rebuild, with dbt targets pointing at it. Staging mirrors production structure and runs the full pipeline on a recent, masked subset or on production data with restricted access, to catch scale and integration problems. Production runs only deployed code, on its own identities and compute, with the SLAs from the observability module. Zero-copy clones and table-format snapshots make staging datasets cheap where the platform supports them.

• Same code, different targets: dbt profiles, orchestrator variables and Terraform var.env select the environment; no if env == 'prod' branches in logic.
• Masked samples for development, generated by a pipeline that applies the governance module's masking rules; refreshed weekly.
• Staging is where backfills and migrations are rehearsed before production.
• Separate cloud accounts or projects per environment, so a staging mistake cannot touch production data or identities.
• Promotion is a deploy, never a manual copy of files or a hand-run notebook.

Cost: attribute, cap, review

Data platforms have three big cost lines: warehouse compute, storage (and the API calls on it), and code compute (Spark clusters, orchestrator workers). The warehouse module covered compute tuning; the platform's job is attribution and control. Tag every resource and query with team and pipeline. Give each domain its own warehouse or reservation so the bill has an owner. Set budgets with alerts at 50, 80 and 100 percent, byte caps on ad hoc access, auto-suspend everywhere, and lifecycle rules on storage. Review the top cost drivers weekly with the owners, and treat an unexplained jump as an incident.

resource "aws_budgets_budget" "data_platform" {
  name         = "data-platform-${var.env}"
  budget_type  = "COST"
  limit_amount = "12000"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  cost_filter {
    name   = "TagKeyValue"
    values = ["user:platform$data"]
  }

  dynamic "notification" {
    for_each = [50, 80, 100]
    content {
      comparison_operator        = "GREATER_THAN"
      threshold                  = notification.value
      threshold_type             = "PERCENTAGE"
      notification_type          = "ACTUAL"
      subscriber_email_addresses = ["data-platform@acme.example"]
    }
  }
}