Storage formats & the lakehouse

Rows versus columns

A row store keeps all the fields of one record together (CSV, JSON lines, a transactional database's heap). It is right for reading or writing whole records: fetch order 1042, insert a new order. A column store keeps all the values of one column together (Parquet, ORC, every analytical warehouse). It is right for scanning a few columns over many rows: total amount for last quarter touches one column and ignores the other forty. Columnar data also compresses far better, because a column holds similar values (many repeated countries, sorted timestamps), and encodings like dictionary and run-length exploit that.

This is why operational databases and analytical systems differ, and why pipelines typically move data from row-oriented sources into columnar files or warehouses. The conversion point is usually the first thing a pipeline does after landing raw data.

Parquet, properly

A Parquet file is split into row groups (typically 64 to 512 MB of rows), each containing one column chunk per column, each chunk in pages. The footer holds the schema and, per column chunk, statistics: min, max, null count, and optionally bloom filters. A reader that wants WHERE ordered_at >= '2026-03-01' inspects the statistics and skips every row group whose max is earlier — predicate pushdown — and reads only the columns the query names — projection pushdown. Sorting data within a file by the columns you filter on makes the statistics tight and the skipping effective.

import pyarrow.parquet as pq

f = pq.ParquetFile("orders/month=2026-03/part-0.parquet")
print(f.schema_arrow)
meta = f.metadata
print("row groups:", meta.num_row_groups, "rows:", meta.num_rows)

col_idx = f.schema_arrow.get_field_index("ordered_at")
for rg in range(meta.num_row_groups):
    col = meta.row_group(rg).column(col_idx)
    st = col.statistics
    print(rg, col.path_in_schema, st.min, st.max, "nulls:", st.null_count,
          "compressed MB:", round(col.total_compressed_size / 1e6, 1))

Write with sensible settings: zstd or snappy compression, row groups of around 128 MB, dictionary encoding on (the default), and timestamps with an explicit unit and timezone. Avoid many tiny files: each file costs a listing and a footer read, and a query over ten thousand 1 MB files is slower than over eighty 128 MB files holding the same data. Compaction — periodically rewriting small files into large ones — is routine maintenance.

Layout in object storage

Object storage (S3, GCS, Azure Blob) is cheap, durable and effectively infinite, with two properties that shape layout: listing is slow and per-object requests have latency, so large objects and shallow prefixes win. Partitioning organises files into prefixes by a column with low cardinality that queries filter on, typically date: orders/dt=2026-03-01/part-0.parquet. A query for one week lists seven prefixes and reads only those files. Over-partitioning (by hour, by customer id) produces thousands of tiny files and slow listings; under-partitioning forces full scans. Aim for partitions of hundreds of MB to a few GB.

s3://acme-lake/
  raw/                              immutable, as received, partitioned by arrival
    orders_api/dt=2026-03-01/hour=13/batch-7f3c.jsonl.gz
  bronze/                           parsed to Parquet, typed, still one row per source record
    orders/dt=2026-03-01/part-0000.parquet
  silver/                           cleaned, deduplicated, conformed; table-format managed
    orders/                         (Iceberg table: metadata/ + data/)
  gold/                             business-level models and aggregates
    daily_revenue_by_region/

rules
  raw:    never modified; retention by policy; replay source for everything below
  files:  128 MB - 1 GB Parquet, zstd; compaction job weekly
  keys:   partition by date of the event, not date of load

The zone names vary (raw/bronze/silver/gold, landing/staging/curated); the idea is constant: an immutable raw layer you can always replay, progressively cleaner layers built by code, and business models at the top. Store the raw data compressed in its original form, because parsing bugs are discovered later and the raw bytes are the only thing that can fix them.

Table formats: the lakehouse

A folder of Parquet files has no transactions: a reader can see a half-written batch, two writers can corrupt each other, a schema change means rewriting everything, and deleting a customer's rows for a privacy request means finding and rewriting the right files by hand. Open table formats — Apache Iceberg, Delta Lake, Apache Hudi — add a metadata layer over the files: a log of snapshots, each listing exactly which files make up the table at that version. Writers commit new snapshots atomically; readers see one consistent snapshot; old snapshots enable time travel and rollback; and the metadata carries the schema, so columns can be added, renamed or reordered without rewriting data.

CREATE TABLE lake.silver.orders (
  order_id    BIGINT,
  customer_id BIGINT,
  ordered_at  TIMESTAMP,
  status      STRING,
  amount      DECIMAL(12,2)
) USING iceberg
PARTITIONED BY (days(ordered_at));        -- hidden partitioning: queries filter on ordered_at, not a dt column

MERGE INTO lake.silver.orders t
USING lake.bronze.orders_batch s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

-- what did the table look like before today's merge?
SELECT COUNT(*) FROM lake.silver.orders TIMESTAMP AS OF '2026-03-01 00:00:00';

-- maintenance: compact small files and expire old snapshots
CALL lake.system.rewrite_data_files('silver.orders');
CALL lake.system.expire_snapshots('silver.orders', TIMESTAMP '2026-02-01 00:00:00');

The result is the lakehouse: warehouse-like tables (ACID, schema, SQL) on lake storage (cheap, open files, many engines). Spark, Trino, Flink, DuckDB and the major cloud warehouses can all read Iceberg tables in the same bucket, which removes the copy-the-data-into-each-tool problem. A catalog (REST, Glue, Nessie, Unity) holds the pointer from table name to current metadata file; it is the one component every engine must agree on.

Warehouse, lake, or lakehouse

A cloud warehouse (BigQuery, Snowflake, Redshift, and similar) is the simplest to run: load data, write SQL, pay for storage and compute, get performance without managing files. A data lake is object storage with files and a query engine on top: cheapest and most flexible, most operational work, historically weakest on consistency. A lakehouse is the lake with table formats, closing most of the gap. The choice is less about technology than about the team: who operates it, which tools must read the data, how much data there is, and whether machine learning and streaming workloads need direct file access.

• Small team, mostly SQL analytics, moderate data: a warehouse, with Parquet exports to a lake for ML and archive.
• Many engines (Spark for ML, Trino for ad hoc, Flink for streams) over the same data: a lakehouse with Iceberg and a shared catalog.
• Regulatory needs for deletion and audit: a table format with row-level delete and snapshot history, whichever platform.
• Whatever you pick: an immutable raw layer, Parquet as the interchange format, and business models defined in code.