The lakehouse pattern, two years later

What a lakehouse actually is

A data warehouse gives reliable, governed tables but usually keeps data in a proprietary store. A data lake keeps everything cheaply in open files but, on its own, offers weak guarantees: concurrent writes can corrupt tables and schemas drift silently.

The lakehouse pattern combines the two. Data stays in open columnar files (typically Parquet) on object storage, and a table format adds ACID transactions, schema enforcement, time travel and metadata that query engines can use to skip irrelevant files. Several engines can then read and write the same tables.

What held up

The decisions that aged well were the boring ones — choices that kept data portable and responsibilities clear.

• An open table format, so storage outlives any single query engine or vendor contract.
• A small number of layers — raw, cleaned and business-ready (often called bronze, silver and gold) — each with a written purpose.
• One named owner per business-ready table, with documented freshness and quality expectations.
• Data contracts at ingestion, so upstream schema changes fail loudly instead of corrupting downstream reports.

What we would not repeat

Streaming ingestion without compaction produces many tiny files that slow every query. Schedule compaction and file-size optimisation from day one, and monitor average file size as a health metric.

Letting every team create tables anywhere led to near-duplicate datasets with conflicting definitions. A catalogue with clear naming rules, access policies and lineage is not optional once more than one team writes data.

Interactive clusters left running overnight were a recurring, avoidable cost. Auto-termination, job clusters for scheduled work and per-team cost reporting fixed most of it.

Choosing a table format

All three major formats are capable; the deciding factor is usually which engines you already run. Verify current feature support against each project's documentation, as it changes quickly.

A migration checklist

Moving from a warehouse or a raw lake is safest when done table by table, with old and new outputs compared before switching consumers.

• Inventory the reports and models that matter, then migrate the tables they depend on first.
• Run old and new pipelines in parallel and reconcile row counts, totals and key metrics.
• Define retention and time-travel limits so storage does not grow without bound.
• Set access policies in the catalogue before granting broad read access.
• Agree a cut-over date per consumer and decommission the old path promptly.