Governance, privacy & access

Catalogue and ownership

The catalogue is the searchable inventory of data assets: tables, columns, dashboards, pipelines, with descriptions, owners, lineage, freshness and classification attached. It exists so that an analyst can find the revenue table without asking, understand its grain without guessing, and know who to contact when it looks wrong. Most of its content is generated: dbt descriptions and tests, orchestrator lineage, warehouse schemas, query logs for popularity. The human part is ownership — every tier-1 asset has a named team — and the definitions, which the modelling module said to write with the DDL.

models:
  - name: dim_customers
    description: One row per customer (SCD2). Natural key customer_id; surrogate customer_key.
    meta:
      owner: crm-data@acme.example
      domain: customer
      classification: confidential
    columns:
      - name: email
        description: Login email; direct identifier.
        meta: {pii: direct, mask: hash}
      - name: date_of_birth
        meta: {pii: indirect, mask: year_only}
      - name: country
        meta: {pii: none}
      - name: segment
        description: Marketing segment, recomputed nightly.

Ownership has to mean something: the owner reviews access requests, answers questions, approves schema changes and is paged for tier-1 incidents. An asset with no owner is a candidate for deletion, not for tolerance. Domains (customer, finance, product) group assets and owners, and a lightweight council of domain owners handles the cross-cutting decisions: shared definitions, classification rules, platform standards.

Classification and personal data

Classification decides how data is handled. A workable scheme has three or four levels — public, internal, confidential, restricted — plus column-level tags for personal data (direct identifiers such as email and name; indirect ones such as date of birth and postcode that identify in combination) and for special categories (health, payment, biometrics) that regulations single out. Classify at ingestion, in the contract or the source definition, so that the tag travels through lineage: a column derived from a PII column is PII unless it has been anonymised.

• Keep a record of processing: which datasets hold personal data, why, for how long, who it is shared with. It is the GDPR requirement and the incident-response inventory in one.
• Minimise: do not land fields you have no purpose for; drop or hash direct identifiers in analytical layers where the raw value is not needed.
• Pseudonymise by default: analytics on a stable hashed id, with the mapping back to identity held in one restricted table.
• Automated scanners can detect likely PII (emails, card numbers, national ids) in untagged columns; run them on new tables and treat hits as classification work.

Access control that follows the data

Access should be granted by role, scoped by classification, and enforced by the platform rather than by convention. Roles map job functions to grants (analyst, finance-analyst, data-engineer, service accounts per pipeline). Row-level security filters rows by the caller's attributes (a regional manager sees their region). Column-level security and dynamic masking hide or transform sensitive columns unless the role is entitled: an analyst sees a hashed email, a support agent sees the full one. Generate the policies from the classification tags so a newly tagged column is protected automatically.

-- column masking: full value for the support role, hashed for everyone else
CREATE OR REPLACE MASKING POLICY mask_email AS (val STRING) RETURNS STRING ->
  CASE WHEN CURRENT_ROLE() IN ('SUPPORT_AGENT', 'DATA_ENGINEER') THEN val
       ELSE SHA2(val, 256) END;
ALTER TABLE analytics.dim_customers MODIFY COLUMN email SET MASKING POLICY mask_email;

-- row access: regional analysts see only their region; central roles see everything
CREATE OR REPLACE ROW ACCESS POLICY region_rows AS (region STRING) RETURNS BOOLEAN ->
  CURRENT_ROLE() IN ('FINANCE_CENTRAL', 'DATA_ENGINEER')
  OR EXISTS (SELECT 1 FROM governance.role_regions r
             WHERE r.role_name = CURRENT_ROLE() AND r.region = region);
ALTER TABLE analytics.fct_orders ADD ROW ACCESS POLICY region_rows ON (region);

-- grants by role, never by user
GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO ROLE ANALYST;
GRANT SELECT ON TABLE analytics.dim_customers_full TO ROLE SUPPORT_AGENT;

Federate roles to the identity provider (groups become roles) so joiners, movers and leavers are handled once, and review access quarterly with the owner approving. Service accounts for pipelines get their own roles with least privilege — the transformation role can write to the marts schema and read raw, and nothing else. Access to raw or restricted zones is exceptional, logged and time-boxed.

Retention and the right to be forgotten

Every data class gets a retention period in the record of processing, and a job enforces it: partitions expire in the warehouse and the lake, raw files age out by lifecycle policy, backups roll off. Where the data must stay for aggregate analytics but the person's identity must not, anonymise instead of delete. Erasure requests (a person asks to be forgotten) are the hard case: you must find every copy of their data — mirror tables, changelogs, event topics, feature stores, exports, backups within a reasonable window — and remove or anonymise it, with a record that you did. Table formats with row-level delete make the tables straightforward; the changelog and the object store need design.

-- 1. identity tables: delete
DELETE FROM lake.silver.customers WHERE customer_id = 1042;
DELETE FROM lake.bronze.customers_changes WHERE COALESCE(after.id, before.id) = 1042;

-- 2. fact tables: keep the row for aggregates, sever the identity
UPDATE lake.silver.orders SET customer_id = NULL, shipping_address = NULL WHERE customer_id = 1042;
UPDATE lake.gold.customer_360 SET email = NULL, name = 'erased' WHERE customer_id = 1042;

-- 3. record it (an auditor and the requester will ask)
INSERT INTO governance.erasure_log (customer_id, requested_at, completed_at, tables_touched, operator)
VALUES (1042, TIMESTAMP '2026-03-02 09:15:00', current_timestamp, 4, 'erasure-job');

-- 4. table-format hygiene: expired snapshots still hold the old rows until they are removed
CALL lake.system.expire_snapshots('silver.customers', current_timestamp - INTERVAL '1' DAY);

Two traps. Time travel and snapshots retain deleted rows until snapshots expire, so erasure must include snapshot expiry within the promised window. And event topics with long retention hold the person's events; either keep personal fields out of events (reference ids only) or use per-key encryption where deleting the key erases the data — crypto-shredding. Design for erasure before the first request; the request always comes.

Audit: who touched what

Governance without an audit trail is a policy with no evidence. Warehouses log every query with user, role, tables, columns and rows scanned; lakehouse catalogs and object stores log access; the orchestrator logs which pipeline wrote which table. Keep these logs centrally with the same retention as the security logs (the Cybersecurity track's logging module), and use them: who queried the restricted schema this month; which service accounts read PII; which tables nobody has read in a year and can be retired; which dashboards drive the most queries and deserve tier-1 care.

SELECT
  user_name,
  obj.value:objectName::STRING     AS table_name,
  col.value:columnName::STRING     AS column_name,
  COUNT(*)                          AS accesses,
  MIN(query_start_time)             AS first_seen,
  MAX(query_start_time)             AS last_seen
FROM snowflake.account_usage.access_history,
     LATERAL FLATTEN(base_objects_accessed) obj,
     LATERAL FLATTEN(obj.value:columns) col
WHERE query_start_time >= DATEADD(month, -1, CURRENT_TIMESTAMP())
  AND obj.value:objectName::STRING IN (SELECT full_name FROM governance.restricted_objects)
GROUP BY 1, 2, 3
ORDER BY accesses DESC;