Building a Governed Data Lakehouse

Introduction

Most data teams building a data lakehouse think of governance, schema conformity, transactional integrity, or versioning as an afterthought. By the time they get to it, there are already three definitions of “active customer,” two pipelines implementing logic for flavours of the same use case, and several dashboards reading from the same source but producing different numbers for the same metric — quietly eroding trust in the entire data platform.

The cost of getting data governance right is tangible. It prevents data swamps, compliance exposure, and AI systems that hallucinate because the data they were trained on was never reliable to begin with.

The decisions you make in the early phase of a greenfield lakehouse project have a lasting impact on the data platform. I learned this by having lived through the consequences of both getting it right early and retrofitting it later. This post is an account of what building a governed lakehouse actually looks like in practice.

Why do organizations need a data lakehouse?

It is important to understand what a data lakehouse is and what problems does it solve.

Earlier, organisations managed their data lifecycle through three separate systems:

a data warehouse for structured analytics
a data lake for raw and unstructured data
separate ML platforms for model development
the BI layer on top of the above systems

Each system had its own access control, audit trail and governance framework. This resulted in an architecture where moving data between systems was quite cumbersome along with the added overhead of maintaining multiple copies of the same data, in different formats, with different freshness guarantees, compounding over time.

The lakehouse pattern combines all of this and aims to simplify the data architecture. It combines the flexibility and cost-effectiveness of a data lake with the reliability and query performance of a data warehouse in a single, open-format and unified architecture. Structured, semi-structured, unstructured, and streaming data: all of these coexist in the same platform. Analytics, ML, and AI workloads run against the same underlying data. And there is one governance layer for all of it.

This is technically possible due to the underlying storage format layer. Traditional data lakes lacked ACID transaction support, which makes it possible for failed jobs to corrupt files, multiple pipelines to write concurrently could compromise data integrity, and schema enforcement was absent by default. Modern open table formats like Delta Lake and Apache Iceberg provide ACID guarantees, schema enforcement, time travel, and audit history on top of Parquet, thereby bringing data warehouse reliability to the flexibility of object storage. That is the foundation everything else is built on.

In Databricks, the Delta Lake’s transaction log, house within the _delta_log directory is the mechanism that implements the above concepts of ACID guarantees, schema enforcement, time travel and audit history.

Every write to a Delta table appends a JSON entry to the log recording exactly what changed, when, and by which operation. This means failed jobs do not corrupt or incorrectly/partially write to the table but instead preserve a clean state, and time travel queries (VERSION AS OF, TIMESTAMP AS OF) are first-class operations which helps users restore previous versions.

However, having a reliable storage format does not automatically mean it is a governed lakehouse.

What is “well-governed” in the context of a data lakehouse?

A well-governed lakehouse:

contains data that can be trusted
is secure
is discoverable
is reliable.

Governance is baked into the architecture from the beginning, not retrofitted afterward.

It also means the organisation is at some stage of a data maturity curve.

In the early stages, the focus is on historical reporting: structured data with batch loads, SQL queries, precanned dashboards.
As maturity increases, teams add streaming data, ML workloads, predictive analytics.
At the most mature end, organisations build AI systems that learn from their data and its surrounding metadata: RAG pipelines, agents, natural language interfaces.

Each stage of that curve makes higher demands on the governance layer beneath it.

Let us walk through how that looks like in practice.

Layered architecture with clear data contracts

The Medallion architecture: the Bronze / Silver / Gold pattern is widely adopted across the industry.

Bronze is raw data, which is stored exactly as it is received, is append-only, and contains ingestion metadata.
Silver is cleaned, conformed, and deduplicated data.
Gold is aggregated and optimised data, ready for consumption.

However, the architecture is not effective if it is inconsistently implemented. The layers work only with a clear contract definition for each layer:

what data enters
in what form
with what guarantees
- what happens when those guarantees are violated.

This is typically effective when enforced via a combination of process and system checks.

In Databricks, Delta Lake schema enforcement makes this esay. Setting mergeSchema to false (the default) means a pipeline writing a column that does not exist in the target table will fail explicitly rather than silently modify or drift the schema to an undesired structure.

Combined with enforceSchema in Auto Loader, this enforces the schema at the Bronze ingestion layer so that upstream schema changes are caught here, rather than being discovered much downstream when the data has already landed in a Gold table.

For the Silver-to-Gold contract specifically, Delta Lake constraints add another enforcement layer:

ALTER TABLE gold.daily_account_revenue
ADD CONSTRAINT revenue_non_negative
CHECK (daily_revenue_usd >= 0);

A constraint violation at write time blocks the operation and surfaces the problem to the pipeline owner immediately. This is enforcement built into the storage layer, not maintained via a separate data quality check or tooling.

In practice, the contract between the Silver and the Gold layers is where ambiguity typically sets in and also where most governance failures can occur. Teams often debate upon what goes into which layer and what the contracts for the layers are. It is important to achieve a clear consensus early on and stick to it.

If a Gold table is rebuilt from a Silver table whose schema was changed without notice, every downstream consumer breaks. Clear contracts mean documented grain, documented SLAs, and a defined breaking-change policy (most importantly involving Standard Operating Procedures around communication to all stakeholders).

The layer boundaries might feel like overhead at the beginning. As the data platform matures, they ensure that the platform is consistently governed as the number of producers and consumers multiplies.

Global naming convention

When several teams are building pipelines independently, naming conventions diverge over time. A column called customer_id in one domain turns out to be a different entity from customer_id in another — same name, different grain, different update cadence. Joins that look correct produce wrong results. Data quality checks pass and produce data inconsistency or corruption downstream. Reports diverge.

The conventions that mattered most in practice: domain-prefixed table names, consistent use of surrogate versus natural keys across the platform, standardised date column naming (created_at, updated_at, effective_from, effective_to), and agreed abbreviation rules for column names in data products.

In Unity Catalog, the three-level namespace (catalog.schema.table) makes it possible to define domain boundaries. A table registered as prod.sales.sales_bookings is architecturally separated from prod.product.events. Users cannot join across domains without explicitly referencing both namespaces. This can be effective only if it is complemented by effective naming conventions, however, it enforces domain separation at the metadata layer.

When data products are declared declaratively, such as the schema, column descriptions, grain, SLAs — naming consistency becomes enforceable rather than advisory. Teams cannot easily drift from a standard that is codified in the data product specification.

On a slightly different note, such problems are usually discovered by Data Quality checks at several points in the pipeline. If a team does not have such checks in place, these problems are reported by customers or business users, which results in delayed problem discovery, wrong decisions being made due to incorrect data and more such probelms. Identifying the source of the problem becomes a tedious task as well, since without the Data Quality checks, it is hard to know where the problem is originating.

Domain ownership

It is not scalable to have a central data team own every table in the lakehouse. Data mesh principles guide such scenarios: the teams closest to the data understand it best, and should own not just the data, but the entire data lifecycle and its governance. In practice, this concept of domain ownership means that the team producing the data is responsible for its quality, its schema evolution, and its SLAs. The platform team provides the guardrails: Unity Catalog access policies, schema validation, lineage tracking and other tooling, but does not own the data itself.

Each table, schema, and catalog in an Unity Catalog has an explicit owner, an user or a group, and permissions are configured and inherited hierarchically. Assigning ownership of the prod.sales schema to the Sales team means they control access grants for their own data products. The platform team retains metastore-level administration without owning the data.

Enforcing these standards prevents teams from building pipelines that do not conform to agreed contracts or using different naming conventions, thereby also preventing them from building tables with fields which cannot be joined reliably across domains.

In practice, the platform team also defines and enforces standards across the metastore — table property requirements, mandatory column tags for PII, required grain and owner properties at table registration. These are enforced via Unity Catalog table properties.

When a team is responsible for their data products:

they write column descriptions that explain business meaning, not just data type.
they define the grain explicitly.
they version their schemas.
they own their SLA and are paged when they miss it.

The difference between a team that owns their data and a team that merely produces it is visible in the quality of the metadata.

Data contracts

A data contract is the explicit agreement between a data producer and its consumers. It mainly covers four things: schema, grain, SLAs, and the breaking-change policy.

Schema: what columns exist, their types, and which are nullable. In Delta Lake, the DESCRIBE DETAIL SQL command and the transaction log provide the schema history, including for prior schema version and the timestamp of each of the changes.

Grain: what one row represents: one event, one account per day, one transaction. This is the most commonly undocumented property, which also means that Data Engineers can easily overlook this and implement inconsistent grains. In Databricks, the grain is best documented as a Unity Catalog table property:

ALTER TABLE silver.transactions
SET TBLPROPERTIES (
  'grain' = 'one row per transaction_id',
  'owner' = 'payments-engineering@company.com',
  'sla_hours' = '4'
);

These properties are queryable via SHOW TBLPROPERTIES and surfaced in the Unity Catalog UI, making the grain a first-class and discoverable attribute.

SLA: when is the data expected to be available and how fresh is it?

Breaking-change policy: what constitutes a breaking change, how much notice consumers receive, and what the migration path looks like.

An example scenario: a producer table’s column type was recently changed from integer to string during a routine pipeline refactor. The downstream Gold table schema data quality check caught it six hours later, after three dashboards had already served incorrect aggregations to business stakeholders. Tracing the root cause took two engineers almost a day. The fix took twenty minutes thereafter. A contract with a breaking-change policy would have prevented the type change from reaching downstream without a version increment and consumer notification.

Governance and access

Data classification precedes access control decisions. Before you can apply access controls, you need to know if the incoming data contains: PII, user generated content, financial data, internal-only metrics, publicly referenceable aggregates.

For instance, Databricks Unity Catalog’s tagging system handles this if used consistently.Tags applied at the column level propagate to downstream derived tables through lineage — meaning a column tagged pii:email in Bronze retains that classification when it appears in Silver and Gold, without requiring manual re-tagging at each layer:

ALTER TABLE bronze.raw_events
ALTER COLUMN user_email
SET TAGS ('pii' = 'email', 'sensitivity' = 'high');

Unity Catalog’s column masking policies then apply automatically based on these tags. So, a role without PII access sees user_email as **** without needing masking code to be implemeted as part of the data pipeline:

CREATE ROW FILTER mask_pii_email ON bronze.raw_events
AS (user) -> CASE
  WHEN is_member('pii-access') THEN true
  ELSE user_email IS NULL
END;

Access controls in most cloud platforms include RBAC (role-based) and ABAC (attribute-based) access controls which provide controls at different granularities. RBAC is table-level and schema-level. ABAC is row-level and column-level. For example, they could be used to mask a PII column for one role while exposing it to another.

PII handling includes masking, but it also includes retention policies, audit logging of who accessed what, and ensuring that PII does not leak through derived columns or aggregations. Governance in regulated environments has to account for derived sensitivity, not just source sensitivity.

Delta Lake’s VACUUM command honours retention thresholds. For PII data with hard deletion requirements, the correct and complete way to delete the data is via the column-level deletion combined with REORG TABLE ... APPLY (PURGE) to physically remove the deleted data from Parquet files. Marking the data as deleted in the transaction log would not suffice.

Data discovery and audits

Data discovery is a hard problem to solve and it becomes challenging over time. This is a problem that partially occurs due to missing or incomplete Data Lineage.

When a number in a board-level dashboard is wrong, the first question is always “where does this come from?” Without an end-to-end data lineage, that question takes days to answer without end-to-end data flow lineage. With lineage, it is a simple query of the tables storing the lineage relationships.

Unity Catalog captures column-level lineage automatically for SQL operations on Databricks: a Gold table metric traced back to the Silver layer’s transformations and back to the Bronze layer’s ingestion, with every intermediate table and column visible in the lineage graph. No additional steps or configuration are required for setting up lineage for SQL workloads.

Python and Spark DataFrame operations require the appropriate writing patterns (using spark.sql() or Delta Lake merge operations) to appear in the lineage graph.

What a catalogue is actually for — versus what people think it is for: most teams treat the data catalogue as a documentation project. A place to write descriptions and tag tables. That is the wrong framing. A catalogue is a runtime artefact. It reflects the live state of the data platform: what tables exist, what they contain, who owns them, when they were last updated, and how they relate to each other. Documentation that lives separately from the pipeline inevitably falls behind the pipeline. Descriptions written at data product creation time — as part of the schema definition — stay current because they are versioned alongside the schema.

Tagging and classification tiers work best when they are mandatory at data product registration time, not optional at consumption time. A sensitivity tag added when a table is first defined is almost always more accurate than one added retroactively when a compliance audit surfaces it.

Data observability

Data observability covers five properties: freshness, volume, distribution, schema, and lineage.

Freshness: is the data as recent as defined in the SLA? On Delta Lake, DESCRIBE HISTORY provides the timestamp of the most recent write operation — making freshness checks queryable without external tooling:

SELECT timestamp, operation, operationParameters
FROM (DESCRIBE HISTORY silver.transactions)
WHERE operation = 'WRITE'
ORDER BY timestamp DESC
LIMIT 1;

Volume: are row counts within expected bounds? Are we recording the % of deviation from the average row counts for the past 30 days? Delta Lake table history also records the number of files added and removed per operation, giving a write time volume signal without a full table scan.

Distribution: are column value distributions stable? A shift in the distribution of a key dimension is often the first signal of an upstream data quality issue

Schema: did the schema change without a corresponding contract update? Delta Lake’s transaction log records every schema change with a timestamp, making it auditable by comparing the current schema against a prior schema version:

from delta.tables import DeltaTable

dt = DeltaTable.forName(spark, "silver.transactions")
history = dt.history(10)

Lineage: can you trace a value in a Gold table back to its source system?

The ownership question matters more than the tooling. A team that owns their data product owns the observability signals for that product.

They set the expected volume numbers.
They define what “freshness” means for their SLA.
They are paged when the distribution shifts.

By making observability declarative, we built it into pipelines without making it another task to be performed later. Expected row count ranges, null rate thresholds, and distribution are defined in the data product specification alongside the schema. The pipeline validates against them at each layer. Violations block promotion from Bronze to Silver. This is not a separate observability system — it is the pipeline enforcing the contract at runtime.

Semantic layer

One governed definition per metric. This is the principle that most data teams agree with consistently.

A semantic layer sits between the physical data model and the consumer — whether that consumer is a BI tool, an analyst running SQL, or an AI agent generating queries. It defines what “revenue” means, what “active user” means, what time zone “today” refers to. One definition, one place, applied consistently.

Without it, different teams build their own definitions in their own tools. The organisation ends up with five different revenue numbers depending on which dashboard you open. This is not a data quality problem, it is a governance problem. The data is correct; the definitions are inconsistent.

The semantic layer is also where the connection to AI readiness becomes concrete. An AI agent generating SQL against your lakehouse will use whatever metric definitions it can infer from column names and table descriptions. If those definitions are inconsistent, the agent produces inconsistent results.

AI-readiness

An LLM querying your lakehouse — whether through a natural language query interface, a RAG pipeline, or a custom agent depends entirely on the quality of your metadata. Column descriptions that say “flag” or “id” are useless to a language model. Column descriptions that say “binary indicator set to 1 when the account has had an active subscription within the last 30 days” are genuinely useful. The difference between a Genie Space that returns accurate results and one that hallucinates is largely determined by the quality of the column-level annotations in Unity Catalog.

AI-readiness means:

descriptions are written for a language model, not a data engineer.
embeddings exist for columns and tables so that semantic search over the data catalogue works.
agent-accessible metadata: what tables exist, what they contain, how they relate — is current and accurate. This is possible if it is maintained at data product definition time rather than retroactively.

A lakehouse without this is not AI-ready. Point an LLM at undocumented tables with opaque column names and you will get confident, plausible, wrong answers. The data quality problem that used to surface as a wrong number in a dashboard now surfaces as a confident hallucination in an AI system that business users trust more, not less, than the dashboard it replaced.

Walking through one example end to end

Follow a single transaction entity from source through to the reporting layer to see how naming, contracts, lineage, and quality checks touch the same record at different points.

Bronze: the transaction arrives from the source system exactly as received as a raw JSON, appended to the Bronze table with an ingested_at timestamp. No transformation. The schema is documented; the grain is one event per row.

Silver: the transaction is cleaned, typed, deduplicated, and conformed to the platform naming convention. txn_id, account_sk, transaction_amount_usd, transaction_ts. The Silver contract specifies grain (one transaction per txn_id), nullable columns, and a freshness SLA of T+4 hours. A row count check validates that Silver contains within 1% of the expected Bronze volume. A null check validates that account_sk is never null.

Gold: the transaction is aggregated to the account-day grain: daily_account_revenue, transaction_count, avg_transaction_value. The Gold contract specifies that “revenue” means sum(transaction_amount_usd) where transaction_status = 'settled'. This definition is registered in the semantic layer. Every downstream consumer such as a BI tool, an analyst SQL query or an AI agent uses the same definition.

Lineage: an Unity Catalog lineage trace from the Gold metric back to Bronze is available. When a reporting discrepancy surfaces, the trace takes minutes, not days.

This is what governance built into the architecture looks like, as opposed to governance implemented afterward.

Ungoverned data platforms

These are some problems arise in a data platform without governance:

Definition drift — the same metric calculated differently by different teams, diverging silently over months
Undocumented PII — personal data in columns that are not tagged, not masked, and not audited; a compliance exposure that nobody knows about until it matters
Broken lineage — a pipeline refactored six months ago but the lineage graph never updated, so the dependency map is wrong
Dashboard divergence — two dashboards showing different numbers for the same metric, both technically correct by their own definitions, neither trustworthy
Small-file proliferation — hundreds of thousands of tiny Parquet files accumulating in a Bronze table with no compaction policy, degrading query performance progressively until someone notices

Conclusion

Based on past experience, implementing the below items up front simplifies your data platform and saves time and effort down the line:

Defining the data contract structure before any team writes their first pipeline.
Agreeing on the naming conventions before any table is created.
Making column descriptions mandatory at schema definition time.
Instrumenting lineage from the start.

The hardest part of building a governed lakehouse is not the tooling. It is the organisational discipline to maintain standards as the number of teams, tables, and consumers grows. Platform guardrails help, but they do not substitute for teams that genuinely own their data products.

Governance is an engineering discipline — built into the architecture, maintained by the teams that own the data, and enforced by the platform.