# Medallion architecture: bronze, silver and gold layers, and when to skip them > Medallion architecture sorts lakehouse tables into bronze, silver and gold layers. What each holds, the SQL between them, the checks, and when to skip it. - URL: https://computese.com/medallion-architecture/ - Author: Duong Quan Nguyen, CEO, Computese - Published: 2026-09-26 - Updated: 2026-10-09 - Topics: Data ## In short - Medallion architecture organizes lakehouse tables into bronze (raw, as received), silver (cleaned, deduplicated, conformed) and gold (business-ready) layers, so quality rises at each hop and later layers can be rebuilt from bronze. - Make every hop safe to rerun: append-only bronze, a merge into silver keyed on the business key with one source row per key, and a rule that only a newer row may overwrite an older one. - Put a gate at each boundary: Delta constraints that fail loudly, quarantine tables for expected bad records, expectations in Lakeflow pipelines, and dbt tests on gold. - The layers are a convention, not a mandate: Databricks calls the pattern a recommended best practice, not a requirement, and small data often fits a warehouse with dbt staging and marts. - Choose the table format and catalog from the engines that must read and write the tables; Delta Lake, Apache Iceberg and Apache Hudi all support upserts, and Delta UniForm lets Iceberg and Hudi clients read Delta tables. Medallion architecture is a data design pattern that sorts lakehouse tables into three layers: bronze holds data as it arrived, silver holds it cleaned, deduplicated and conformed, and gold holds business-ready tables for reports, models and APIs. Quality rises at each hop, and because bronze keeps all historical data, the later layers can be rebuilt. The examples run on Delta Lake and Spark. Gold-layer modelling has its own guide, the [star schema](https://computese.com/star-schema/), shared metric definitions live in the [semantic layer](https://computese.com/semantic-layer/), and our overview of [big data analytics](https://computese.com/the-power-of-big-data-analytics/) covers the wider choice between a data lake, a warehouse and a lakehouse. ## Where does the medallion pattern come from? A [Databricks blog post dated August 14, 2019](https://www.databricks.com/blog/2019/08/14/productionizing-machine-learning-with-delta-lake.html) describes tables for data ingestion ("Bronze"), transformation and feature engineering ("Silver"), and machine learning training or prediction ("Gold"), and calls the set a multi-hop architecture; the post does not use the word medallion. The current [Databricks documentation](https://docs.databricks.com/aws/en/lakehouse/medallion) treats the names as synonyms: a way to organize data logically so that structure and quality improve step by step from bronze to silver to gold. Databricks calls following the pattern "a recommended best practice but not a requirement", and the Delta Lake project's [medallion explainer](https://delta.io/blog/delta-lake-medallion-architecture/) describes it as a name for a tier-based architecture to follow in a flexible manner: a convention, not a mandate. Other platforms have adopted it; Microsoft calls medallion architecture [the recommended design approach for Fabric](https://learn.microsoft.com/en-us/fabric/onelake/onelake-medallion-lakehouse-architecture). The pattern assumes a lakehouse underneath, which the [CIDR 2021 paper](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) characterizes by open direct-access data formats such as Apache Parquet and ORC, first-class support for machine learning and data science workloads, and state-of-the-art performance. ## What does each layer hold? Databricks and Microsoft describe the same three-way split with different adjectives: bronze (raw), silver (validated or enriched) and gold (enriched or curated). The words differ and the roles do not. The table pairs the roles they document with the load method and checks this guide recommends. | Layer | Purpose | How it is written | Checks | Who reads it | | ------ | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Bronze | Raw data as received, kept for replay and audit | Append-only; original values, mostly as strings; metadata columns for source and load time | Minimal: capture the schema and counts, keep malformed rows | Data engineers, data operations, compliance and audit teams | | Silver | One validated, non-aggregated row per record | Incremental upsert on the business key; SCD Type 2 where history matters | Type casts, null and range checks, deduplication, constraints, quarantine | Data engineers, data analysts, data scientists | | Gold | Tables modelled for business questions: facts, dimensions, aggregates, feature tables | Rebuilt or refreshed incrementally from silver | Tests on grain and uniqueness, reconciliation to source totals | Business analysts and BI developers, data scientists and ML engineers, executives, operational teams | ### Bronze: keep what arrived Bronze holds raw, unvalidated data. [Databricks](https://docs.databricks.com/aws/en/lakehouse/medallion) lists its traits: it keeps the raw state of the source in its original format, is appended incrementally and grows over time, is the single source of truth, and enables reprocessing and auditing by retaining all historical data. [Microsoft's Fabric guidance](https://learn.microsoft.com/en-us/fabric/onelake/onelake-medallion-lakehouse-architecture) is blunter: store everything exactly as it arrives, with no changes allowed. Validation in bronze is minimal on purpose: Databricks recommends storing most fields as strings, VARIANT or binary to protect against unexpected schema changes, plus metadata columns for provenance such as the source file name. It also says bronze is meant for the workloads that build silver, not for analysts and data scientists. ### Silver: one validated row per record Silver is where cleanup and validation happen. Databricks lists the operations: schema enforcement, handling nulls and missing values, deduplication, resolving out-of-order and late-arriving data, data quality checks, schema evolution, type casting and joins. It advises against writing to silver directly from ingestion: schema changes and corrupt records in the sources would turn into load failures. ![Three trays in a row hold cards that get tidier from left to right: a loose heap, sorted rows, then a few summary sheets. An orange funnel stands between the first two trays.](https://computese.com/images/blog/medallion-architecture/layers.3c338db75e-1536.webp) *The gate sits between bronze and silver: every later layer starts from records that already passed it.* Databricks's bar for silver: it "should always include at least one validated, non-aggregated representation of each record", so keep the grain of the source: one row per order, per payment, per customer version. Conform keys here, so a customer has one id from the shop, the payments provider and the CRM alike. Where attribute history matters, such as the address a customer had when an order shipped, keep versions in a slowly changing dimension of Type 2; the [Delta Lake merge documentation](https://docs.delta.io/delta-update/) shows the pattern. ### Gold: shaped for a question Gold holds what people and applications query: dimensional models, aggregates and feature tables. Databricks describes gold data as aggregated and filtered for specific needs, mapped to business functions, with some customers keeping several gold layers for domains such as HR, finance and IT. Gold reads from silver, not bronze: the [2019 post](https://www.databricks.com/blog/2019/08/14/productionizing-machine-learning-with-delta-lake.html) warns that connecting every gold table straight to raw data makes each business unit repeat the same ETL and invites diverging metric definitions. The Delta Lake project notes exceptions, such as a gold table that [joins a bronze table and a silver table](https://delta.io/blog/delta-lake-medallion-architecture/). ## How do orders and payments move through the layers? The example is an online shop. Orders arrive as change events from the shop's database, and payments arrive from a payments API. Every value is illustrative. The SQL and Python below were run on Apache Spark 4.2.0 with Delta Lake 4.4.1 in local mode. ### Bronze: land the events Bronze keeps the business columns as strings and adds three metadata columns. `_event_id` is whatever uniquely identifies a received event, such as a change log position or a file name plus a row number. ```sql CREATE TABLE IF NOT EXISTS bronze.orders_cdc ( order_id STRING, customer_id STRING, status STRING, total STRING, currency STRING, created_at STRING, updated_at STRING, _event_id STRING, _ingested_at TIMESTAMP, _source STRING ) USING DELTA; ``` A first delivery puts six rows in bronze for four orders, all dated 2026-10-07. Order 1001 appears three times: pending, then paid, then a duplicate of the pending event that arrives late. Order 1002 carries a decimal comma in its total. | `_event_id` | `order_id` | `status` | `total` | `updated_at` | | ----------- | ---------- | --------- | ------- | ------------ | | evt-0001 | 1001 | pending | 129.90 | 09:00:05 | | evt-0002 | 1001 | paid | 129.90 | 09:02:11 | | evt-0003 | 1001 | pending | 129.90 | 09:00:05 | | evt-0004 | 1002 | paid | 12,50 | 10:15:09 | | evt-0005 | 1003 | paid | 49.00 | 11:30:04 | | evt-0006 | 1004 | cancelled | 80.00 | 12:05:00 | ### Silver: type, deduplicate, quarantine The silver load makes four decisions, and each one shows in the code. 1. **Type without failing.** `try_cast` returns NULL for a value it cannot convert. In Spark 4.2, [ANSI mode is on by default](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html), so a plain `CAST('12,50' AS DECIMAL(12,2))` raises an error instead; in the local run, swapping `try_cast` for `CAST` failed the whole streaming query on that one value. 2. **Divert, do not drop.** Rows whose key, timestamp or amount cannot be typed go to a quarantine table with a reason and a timestamp; the insert is a merge keyed on `_event_id`, so a restart cannot insert the same reject twice. 3. **One row per key.** `ROW_NUMBER()` keeps the latest valid event per order; a merge needs at most one source row per target row. 4. **Only newer wins.** `WHEN MATCHED` updates a row only when the incoming `updated_at` is later, so a late duplicate cannot undo a newer state. ```sql MERGE INTO quarantine.orders_rejected AS q USING ( SELECT *, 'order_id, updated_at or total is not valid' AS reason, current_timestamp() AS rejected_at FROM orders_batch WHERE try_cast(order_id AS BIGINT) IS NULL OR try_cast(updated_at AS TIMESTAMP) IS NULL OR try_cast(total AS DECIMAL(12,2)) IS NULL ) AS r ON q._event_id = r._event_id WHEN NOT MATCHED THEN INSERT *; ``` ```sql CREATE OR REPLACE TEMP VIEW orders_latest AS WITH typed AS ( SELECT try_cast(order_id AS BIGINT) AS order_id, try_cast(customer_id AS BIGINT) AS customer_id, status, try_cast(total AS DECIMAL(12,2)) AS total_amount, currency, try_cast(created_at AS TIMESTAMP) AS created_at, try_cast(updated_at AS TIMESTAMP) AS updated_at, _ingested_at FROM orders_batch ), ranked AS ( SELECT *, ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY updated_at DESC, _ingested_at DESC) AS rn FROM typed WHERE order_id IS NOT NULL AND updated_at IS NOT NULL AND total_amount IS NOT NULL ) SELECT order_id, customer_id, status, total_amount, currency, created_at, updated_at FROM ranked WHERE rn = 1; MERGE INTO silver.orders AS t USING orders_latest AS s ON t.order_id = s.order_id WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *; ``` The job reads bronze as a stream, which Databricks recommends for append-only sources, and runs the three statements per micro-batch, held in `QUARANTINE_SQL`, `LATEST_SQL` and `MERGE_SQL`. ```python def load_silver(batch_df, batch_id): batch_df.createOrReplaceTempView("orders_batch") for statement in (QUARANTINE_SQL, LATEST_SQL, MERGE_SQL): batch_df.sparkSession.sql(statement) ( spark.readStream.table("bronze.orders_cdc") .writeStream.foreachBatch(load_silver) .option("checkpointLocation", "/checkpoints/silver_orders") .trigger(availableNow=True) .start() .awaitTermination() ) ``` After the first run, silver holds order 1001 (paid, 129.90), order 1003 (paid, 49.00) and order 1004 (cancelled, 80.00), and quarantine holds order 1002 with its `12,50` total. A later delivery brings a stale copy of the pending event for order 1001 and a corrected event for order 1002 with a total of 12.50. On the next run order 1002 enters silver and order 1001 stays paid. Applying the first micro-batch twice, or restarting after it with no new data, left silver at three rows and quarantine at one. ### Gold: model for the question Gold needs payments as well as orders. Silver also holds a payments table, loaded the same way: a 129.90 capture and a 20.00 refund for order 1001, and a 49.00 capture for order 1003. ```sql CREATE OR REPLACE TABLE gold.fct_orders USING DELTA AS WITH paid AS ( SELECT order_id, SUM(CASE WHEN kind = 'capture' THEN amount ELSE 0 END) AS amount_captured, SUM(CASE WHEN kind = 'refund' THEN amount ELSE 0 END) AS amount_refunded FROM silver.payments GROUP BY order_id ) SELECT o.order_id, o.customer_id, CAST(o.created_at AS DATE) AS order_date, o.status AS order_status, o.currency, o.total_amount, COALESCE(p.amount_captured, 0) AS amount_captured, COALESCE(p.amount_refunded, 0) AS amount_refunded FROM silver.orders AS o LEFT JOIN paid AS p ON p.order_id = o.order_id; CREATE OR REPLACE TABLE gold.daily_revenue USING DELTA AS SELECT order_date, currency, COUNT_IF(amount_captured > 0) AS paid_orders, SUM(amount_captured - amount_refunded) AS net_revenue FROM gold.fct_orders WHERE order_status <> 'cancelled' GROUP BY order_date, currency; ``` `fct_orders` has one row per order. `daily_revenue` has one row for 2026-10-07 in CAD, with 2 paid orders and net revenue of 158.90, which is 129.90 - 20.00 + 49.00. The cancelled order is excluded, and order 1002 is marked paid but has no captured payment yet, so it adds nothing until the payment arrives. A reconciliation test would flag that gap. Rebuilding gold in full with `CREATE OR REPLACE TABLE` is fine while silver is small. When the rebuild gets slow, refresh from silver's change feed (next section) or use materialized views: Databricks shows a [gold aggregate as one](https://docs.databricks.com/aws/en/lakehouse/medallion), and Fabric's [materialized lake views](https://learn.microsoft.com/en-us/fabric/onelake/onelake-medallion-lakehouse-architecture) pick incremental, full or no refresh per view. ## How do you make a rerun safe? Every pipeline gets rerun: a job restarts, a bug is fixed, a source resends a file. A safe rerun changes nothing that should not change. The example guards against four specific failures. | What happens | Without a guard | Guard in the example | | --------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------- | | A restart applies the same micro-batch twice | Duplicate rows and double-counted revenue | `MERGE` keyed on `order_id`, and a quarantine insert keyed on `_event_id` | | One batch holds two events for the same order | The merge can fail, because it is unclear which source row to use | `ROW_NUMBER()` keeps the latest event per order | | An old event arrives after a newer one | Silver goes back in time | `WHEN MATCHED AND s.updated_at > t.updated_at` | | A value cannot be typed | One malformed value stops the streaming query | `try_cast`, then quarantine | > [!WARNING] > A merge source must hold one row per key. [Delta Lake](https://docs.delta.io/delta-update/) can fail a merge when several source rows match the same target row, and the [Apache Iceberg documentation](https://iceberg.apache.org/docs/latest/spark-writes/) says only one source record can update any given target row, or an error is thrown. Reduce the source to the latest row per key first, as `orders_latest` does. A bank's regulatory data proof of concept met the same problem from the other side: its raw layer was append-only, and without de-duplication to the latest version above it, the first correction would have been counted twice (see the [case study](https://computese.com/work/banking-regulatory-data/)). The `availableNow` trigger suits scheduled runs: [Spark's documentation](https://spark.apache.org/docs/latest/streaming/apis-on-dataframes-and-datasets.html) says it processes all available data at the time of the run, in possibly several micro-batches, then stops on its own, so the compute can shut down between runs. Databricks lists triggered incremental ingestion as lower cost and higher latency than a continuous stream, so pick the slowest schedule the business accepts. `foreachBatch` hands your function each micro-batch and its id, and Delta Lake warns that restarts can reapply a batch, so the merge inside it must be idempotent. Spark can also drop duplicates inside the stream. A watermark sets how late a duplicate may arrive and lets Spark discard old state; without one, the query stores state for all past records. A merge on the business key has no such time window, because it compares against the table itself, so treat stream-level deduplication as an optimization and the merge as the safeguard. To refresh gold from only what changed, turn on the [change data feed](https://docs.delta.io/delta-change-data-feed/). It is off by default and records only changes made after it is enabled, and it suits silver and gold tables because ETL can process just the row-level changes. ```sql ALTER TABLE silver.orders SET TBLPROPERTIES (delta.enableChangeDataFeed = true); -- the second argument is the first table version to read SELECT order_id, status, _change_type, _commit_version FROM table_changes('silver.orders', 1); ``` The strict `>` in the merge matters here. In the local run, the same merge written with `>=` added an `update_preimage` and an `update_postimage` row to the feed for every identical replay, although nothing had changed. With `>`, replays added nothing. Reprocessing is the payoff for keeping bronze: when a silver rule turns out to be wrong, fix the SQL and replay bronze through it. In the example run, deleting silver and merging all of bronze in one pass rebuilt the same four orders. On a real table, rebuild into a new table, compare counts and totals, then swap. How far back a replay reaches depends on bronze retention, so set it on purpose. ![A raw tray feeds a clean tray along a straight arrow with a gear on it. A curved orange arrow loops from the raw tray back over the gear into the clean tray, the replay after a fix.](https://computese.com/images/blog/medallion-architecture/replay.29617ca928-1536.webp) *A bug fixed in silver is repaired by replaying bronze, so the source system is never asked to resend anything.* ## Which checks stop bad data between layers? A layer boundary is a good place for a check, because the next layer should only see data that passed it. Four mechanisms cover most cases, and they differ in what happens to a failing row. | Mechanism | Runs | What happens to a failing row | Best for | | ---------------------------------------- | ---------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------- | | Delta `CHECK` and `NOT NULL` constraints | On every write to the table | The write fails with an invariant violation | Rules that must never break, such as a non-negative total | | Lakeflow pipeline expectations | Per record, inside a pipeline | Warn (keep and count), drop (remove and count) or fail the update | Rules you want measured on every run | | Quarantine table | In your own SQL | The row is diverted with a reason and a timestamp | Expected, fixable problems such as a malformed amount | | dbt data tests | After a build, on the built tables | The test query returns the failing rows; zero rows means a pass | Grain, uniqueness and relationships in gold | Delta Lake [constraints](https://docs.delta.io/delta-constraints/) are the built-in gate. Adding one with `ALTER TABLE ... ADD CONSTRAINT` first verifies that all existing rows satisfy it, and it upgrades the table's writer protocol version, so confirm every engine writing to the table supports the new protocol first. ```sql ALTER TABLE silver.orders ADD CONSTRAINT total_not_negative CHECK (total_amount >= 0); ``` In the local run, an insert of a negative total was rejected with a constraint violation error. Lakeflow pipelines is the current name of the product formerly known as Delta Live Tables; Databricks's [rename notice](https://docs.databricks.com/aws/en/ldp/where-is-dlt) says no migration is needed, and the new `dp` Python API adds compatibility with Apache Spark Declarative Pipelines from Spark 4.1. Quality rules there are [expectations](https://docs.databricks.com/aws/en/ldp/expectations), with three actions: warn, the default, writes invalid records to the target; drop removes them before the write and logs the count; fail stops the update until someone intervenes. The same page describes a quarantine pattern for cases where neither dropping nor failing is acceptable. ```sql CREATE OR REFRESH STREAMING TABLE silver_orders ( CONSTRAINT valid_total EXPECT (total_amount >= 0) ON VIOLATION DROP ROW ) AS SELECT * FROM STREAM(bronze_orders_typed); ``` An expectation cannot call a custom Python function, call an external service or use a subquery on another table, so a rule such as "the customer exists" belongs in a join in silver or in a dbt test. [dbt data tests](https://docs.getdbt.com/docs/build/data-tests) sit in YAML next to the model and run after the build. dbt ships four generic tests, `unique`, `not_null`, `accepted_values` and `relationships`, and a test passes when its query returns zero failing rows. ```yaml models: - name: fct_orders columns: - name: order_id data_tests: - unique - not_null - name: order_status data_tests: - accepted_values: arguments: values: [pending, paid, cancelled] ``` The `arguments:` key needs dbt 1.10.5 or later; older versions set the values as a top-level property of the test. A singular test (a SQL file returning failing rows) can reconcile gold totals with silver. For the wider practice of tests, contracts and freshness checks, see our guide to [building data analytics software](https://computese.com/building-data-analytics-software/). The quarantine table is a plain Delta table holding the rejected row, a reason and a timestamp. Two habits make it useful: alert when the reject count rises, and give every reject a path back. When the source corrects a record, the corrected event arrives as a new bronze row and flows through, as order 1002 did. ## Which table format and catalog should the layers use? The table format decides which engines can read the layers and what upkeep they need. All three open formats support row-level upserts, so the pattern works on each. | Format | What the documentation says | Row-level upserts | | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | Delta Lake | Supports ACID transactions and stores data as Parquet files with a transaction log; the [default format in a Fabric lakehouse](https://learn.microsoft.com/en-us/fabric/onelake/onelake-medallion-lakehouse-architecture) | [`MERGE`](https://docs.delta.io/delta-update/) | | [Apache Iceberg](https://iceberg.apache.org/) | The open table format for analytic datasets, which lets engines such as Spark, Trino and Flink safely work with the same tables at the same time | [`MERGE INTO`](https://iceberg.apache.org/docs/latest/spark-writes/), recommended over `INSERT OVERWRITE` because it replaces only the affected data files | | [Apache Hudi](https://hudi.apache.org/docs/overview) | An open table format purpose-built for high-performance writes on incremental data pipelines | Efficient upserts and deletes | You do not have to pick one format for every consumer. [Delta Universal Format (UniForm)](https://docs.delta.io/delta-uniform/) lets Iceberg and Hudi clients read a Delta table: it generates the metadata asynchronously, and a single copy of the data files serves clients of all formats. It works because all three formats are Parquet data files plus a metadata layer, and it has requirements such as column mapping, so test it on one table first. Our piece on [data compression](https://computese.com/revolutionary-data-compression-algorithm/) covers how Parquet shrinks data. The catalog is the second decision. Status of the main options as of October 2026: | Catalog | Status as of October 2026 | Worth knowing | | ------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | [Unity Catalog](https://docs.unitycatalog.io/) (open source) | Apache 2.0 licence; its documentation, last updated in November 2025, calls it a sandbox project with the LF AI and Data Foundation | Compatible with the Apache Hive metastore API and the Apache Iceberg REST catalog API | | [Apache Polaris](https://news.apache.org/foundation/entry/the-apache-software-foundation-graduates-two-open-source-projects-from-incubator) | An Apache Top-Level Project since March 5, 2026 | An open source catalog for Apache Iceberg that implements Iceberg's REST API, for engines including Spark, Flink and Trino | | [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/connect-glu-iceberg-rest.html) | Offers an Iceberg REST endpoint | Supports the operations in the Iceberg REST specification, and Iceberg table specs v1 and v2 (v2 by default) | | [Snowflake](https://docs.snowflake.com/en/user-guide/tables-iceberg) | Iceberg tables with Snowflake-managed storage, or cloud storage that you manage | Snowflake can be the Iceberg catalog, or connect to an external one through a catalog integration | Decide from the engines that must read and write the tables, then choose the catalog those engines support. Two of our cases show the range: [a proof of concept on a bank's Cloudera platform](https://computese.com/work/banking-regulatory-data/) keeps its raw tables in Apache Iceberg, and [a healthcare data lake on AWS](https://computese.com/work/healthcare-data-lake/) catalogs datasets in Amazon S3 with AWS Glue, queries them with Amazon Athena and runs serverless. ## How does the pattern look on Databricks, Fabric and dbt? | Platform | How the layers are expressed | Worth knowing | | ---------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | Databricks | Each layer is a schema in one catalog (`ops.bronze`, `ops.silver` and `ops.gold` in the documentation's example) | Pipelines or Structured Streaming jobs move data between layers; how often they run trades cost against latency | | Microsoft Fabric | A lakehouse per layer, or lakehouses for bronze and silver and a warehouse for gold | Each lakehouse preferably in its own workspace; shortcuts instead of copies in bronze | | dbt | Staging, intermediate and marts models | The guide moves data from source-conformed to business-conformed and does not use the word medallion | Fabric offers two [deployment patterns](https://learn.microsoft.com/en-us/fabric/onelake/onelake-medallion-lakehouse-architecture): each layer as a lakehouse, with business users reading through the SQL analytics endpoint, or bronze and silver as lakehouses and gold as a data warehouse. Microsoft recommends one workspace per lakehouse, for more control and better governance at the layer level, and shortcuts in bronze instead of copies when the source is in OneLake, ADLS Gen2, Amazon S3 or Google storage. Materialized lake views declare silver and gold in SQL while Fabric handles dependencies and picks the refresh. File size guidance varies by layer: 128 MB to 1 GB, smaller in bronze, larger in gold. dbt's [structure guide](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview) describes a path from source-conformed to business-conformed data through staging, intermediate and marts models. Its [staging page](https://docs.getdbt.com/best-practices/how-we-structure/2-staging) asks for a one-to-one relationship between staging models and source tables. A common mapping, which is a convention and not dbt's wording, is that bronze is what dbt reads as sources, staging and intermediate models play the silver role, and marts play the gold role. ## Who may read each layer, and how do you trace a number? Access should follow the layers; the table lists Databricks's intended readers. Whatever personal data a source holds lands in bronze untouched, so bronze is the layer to lock down. Give analysts silver or gold, and decide at the silver step which identifiers are dropped, masked or moved to a restricted table. ![A locked filing cabinet with an orange padlock holds raw records. Only cards passed through a filter, with personal details blacked out, reach the open trays on the right.](https://computese.com/images/blog/medallion-architecture/cabinet.89132a3301-1536.webp) *Raw records stay behind the lock; analysts and applications read the cleaned copy that crosses the filter.* Make the layer boundary a permission boundary. In Databricks's example each layer is a separate schema in one catalog, and the Delta Lake project shows the same namespacing, so a name such as `something_cool.bronze.raw_transactions` states its layer. If models or assistants query your data, treat them like any other reader and point them at gold: our guide to [generative AI for databases](https://computese.com/generative-ai-for-databases/) covers safe, read-only access. Plan deletion for each layer. Removing a person's rows from silver does not remove them from bronze or from earlier table versions. In Delta Lake, [`VACUUM`](https://docs.delta.io/delta-utility/) removes files that are no longer referenced and are older than the retention threshold, 7 days by default; it is not triggered automatically, and time travel to versions older than the retention period is lost after it runs. The change data feed follows the table's retention, so `VACUUM` deletes that history too. Lineage answers where a figure came from. [OpenLineage](https://openlineage.io/docs/) is an open framework for collecting and analyzing lineage; its model records datasets, jobs and runs, with integrations for Apache Spark, dbt and Apache Airflow. ## When is the medallion pattern the wrong choice? The pattern is a default, and it has costs: each materialized layer stores a copy of the data and runs a job to build it. These are the failures to look for, with the fix for each. | Symptom | Likely cause | Fix | | ---------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------ | | Silver has more rows than the source | Replays append, or the merge uses the wrong key | Merge on the business key and keep one source row per key | | Two dashboards show different revenue | Several gold tables each define revenue | Build gold from silver at one grain and define each metric once | | One malformed value stops the pipeline | A strict cast in a Spark 4 job, where ANSI mode is on by default | `try_cast` plus a quarantine table | | A bug cannot be replayed away | Bronze was edited, shortened or dropped | Keep bronze append-only and set its retention to the replay window | | Storage and compute grow with no new use | A layer that only copies the one before it | Remove the layer or make it a view | **Three copies with no purpose.** Name the job of each layer: bronze preserves and replays, silver validates and conforms, gold models for consumption. A layer with no job should be merged into its neighbour or turned into a view. dbt's staging guide takes that line: staging models should typically be materialized as views, because consumers do not query them and a copy would waste warehouse space. **Layers as physical mandates.** Databricks defines the pattern as a logical organization of data, and the Delta Lake project says to follow it flexibly. Three layers can be three schemas in one catalog, or three workspaces on Fabric. Splitting data across more accounts and pipelines than your team can run is a cost, not part of the pattern. **Gold sprawl.** Several gold layers are fine when each serves a distinct domain, as in Databricks's HR, finance and IT example. The failure is several gold tables for the same entity, each with its own revenue formula, the diverging-metrics problem the 2019 post warned about. Keep gold at the grain of a business process, model it as a [star schema](https://computese.com/star-schema/), and define each metric once in a [semantic layer](https://computese.com/semantic-layer/). **Small data.** If the data fits comfortably in a warehouse and one team owns the pipeline, raw tables plus dbt staging and marts models give the same discipline without a lakehouse. As our data platform guidance puts it, a warehouse suits structured reporting at modest volume and a lakehouse suits mixed data, larger volumes and machine learning on the same storage; a sound warehouse earns tests, definitions and monitoring, not a migration. Our overview of [big data analytics](https://computese.com/the-power-of-big-data-analytics/) explains when big data tools are not needed. **Latency and cost per hop.** Databricks lists continuous incremental ingestion as higher cost and lower latency than triggered ingestion. Every hop adds one or the other. If one report needs data within seconds, serve it from a purpose-built path instead of pushing every table through three continuous hops. **Year two.** Upkeep arrives after launch. `VACUUM` does not run on its own, and a Delta table's history grows until retention removes it. One lesson from the bank proof of concept: dropping and recreating an Iceberg table erases its snapshots, so write with insert-overwrite instead. Schedule maintenance when you build the pipeline, not when the storage bill arrives. ## How do you roll it out? Start narrow and add layers as each earns its place. 1. **Pick one question and trace it back.** Choose one report or model that people rely on or dispute, and list the sources behind each number. That is your first gold table and its inputs. 2. **Land the sources in bronze.** Append-only, original values as strings, metadata columns, and a retention period that covers your replay window. No cleanup. 3. **Build silver for those sources.** Types, conformed keys, one row per record, a guarded merge, a quarantine table. Add `CHECK` constraints for rules that must never break. 4. **Build the one gold table and test it.** Fix the grain, add uniqueness, accepted-value and relationship tests, and reconcile it to a number the business already trusts. 5. **Rehearse a replay.** Rebuild silver from bronze into a scratch schema, compare counts and totals with production, and time the run. 6. **Schedule the upkeep.** Choose the slowest trigger the use case accepts, schedule `VACUUM` or snapshot expiry, set file sizes by layer, and alert on quarantine counts and freshness. 7. **Lock down and trace.** Give each layer its own access rules, keep raw personal data in bronze only, and capture lineage. 8. **Add the next source.** Repeat steps 2 to 4, and add a layer only when you can name its job. > [!TIP] > Rehearse the replay before you need it. Record how long a full rebuild takes and which bronze rows it reads; that figure sets how long bronze must keep data. If you want a second pair of eyes on the design, our [data platform service](https://computese.com/services/data-platform/) starts with a data platform assessment: a review of your sources, reports and current pipelines, ending in a target architecture and a first use case worth building. The bank proof of concept lists medallion layers and SCD Type 2 among its data modelling techniques and is the closest example of this guide's design on real regulatory data. ## Key terms - **Medallion architecture**: A data design pattern that organizes lakehouse tables into bronze, silver and gold layers so that structure and quality improve at each step. Also called a multi-hop architecture. - **Lakehouse**: A data architecture that keeps data in open formats on object storage and serves both BI and machine learning from it, with a table format such as Delta Lake or Apache Iceberg on top. - **Bronze layer**: The first layer: raw data as received from the sources, appended and never edited, with metadata such as source and load time. It is the replay point for everything built on top. - **Silver layer**: The validated layer: records are typed, deduplicated, conformed and checked, at the grain of the source (one row per order, not per day). Analysts and data scientists can read it. - **Gold layer**: The consumption layer: tables modelled for business questions, such as fact tables, dimensions, aggregates and feature tables, built from silver. - **Idempotent load**: A load that can run twice on the same input and leave the same result, so a restart or a replay does not create duplicates or double counts. - **MERGE (upsert)**: A SQL statement that matches source rows to a table on a key, then updates the matches and inserts the rest. Delta Lake and Apache Iceberg both support it. - **Change data feed**: A Delta Lake feature that records row-level changes between table versions (inserts, updates and deletes) so downstream tables can process only what changed. It is off by default. - **Quarantine table**: A side table that holds rows that failed validation, with the reason and the time, so they can be fixed and reprocessed instead of being dropped or breaking the load. - **Open table format**: A specification such as Delta Lake, Apache Iceberg or Apache Hudi that adds a table layer to data files in open formats, so engines can read and update the same tables. ## Common questions ### What is medallion architecture in simple terms? It is a way of organizing lakehouse tables into three quality tiers. Bronze keeps data as received, silver holds cleaned and deduplicated records, and gold holds tables modelled for reports, models and applications. Databricks documents it as a recommended best practice, not a requirement. ### Is medallion architecture only for Databricks? No. Microsoft's Fabric documentation calls it the recommended design approach for Fabric, and the layers can be built from Delta Lake or Apache Iceberg tables. dbt's staging, intermediate and marts layers follow a similar path from source data to business models, although dbt's structure guide does not use the word medallion. ### Do I need all three layers? No. Keep a layer only if you can name its job. The Delta Lake project says to follow the pattern in a flexible manner, and a small dataset that fits a warehouse is often served well by raw tables plus dbt staging and marts models. ### What is the difference between a data lakehouse and a data warehouse? A warehouse suits structured reporting at modest volume; a lakehouse suits mixed data, larger volumes and machine learning on the same storage. The CIDR 2021 paper characterizes a lakehouse by open direct-access data formats, first-class support for machine learning and state-of-the-art performance. Many teams start with a warehouse and grow into open table formats. ### Can gold tables read straight from bronze? They can, but the default is not to. Databricks's 2019 description of the pattern says that connecting gold tables directly to bronze would make each business unit repeat the same ETL and could produce diverging metrics. The Delta Lake project notes that a gold table can join a bronze table and a silver table, so treat the rule as a default. ### What should happen to records that fail validation? Hold them instead of silently dropping them. Write them to a quarantine table with a reason and a timestamp, alert when the count rises, fix the rule or the source, and let the corrected record flow through. Lakeflow pipeline expectations can warn, drop or fail, and Databricks documents a quarantine pattern for cases where neither dropping nor failing is acceptable. ### How long should bronze data be kept? As long as you may need to replay or audit it, which is a business decision. Table history is separate: in Delta Lake, VACUUM removes unreferenced files older than the retention threshold, 7 days by default, and time travel to older versions is lost after it runs. ## Sources 1. [Productionizing Machine Learning with Delta Lake](https://www.databricks.com/blog/2019/08/14/productionizing-machine-learning-with-delta-lake.html), Databricks Blog (August 14, 2019) 2. [What is the medallion lakehouse architecture?](https://docs.databricks.com/aws/en/lakehouse/medallion), Databricks documentation 3. [Building the Medallion Architecture with Delta Lake](https://delta.io/blog/delta-lake-medallion-architecture/), Delta Lake project 4. [Understand medallion architecture for Fabric with OneLake](https://learn.microsoft.com/en-us/fabric/onelake/onelake-medallion-lakehouse-architecture), Microsoft Learn 5. [Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf), CIDR 2021 6. [Table deletes, updates, and merges](https://docs.delta.io/delta-update/), Delta Lake documentation 7. [ANSI Compliance (Spark 4.2.0)](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html), Apache Spark documentation 8. [Writes (Spark, MERGE INTO)](https://iceberg.apache.org/docs/latest/spark-writes/), Apache Iceberg documentation 9. [Apache Iceberg](https://iceberg.apache.org/), Apache Iceberg project 10. [Structured Streaming Programming Guide (Spark 4.2.0)](https://spark.apache.org/docs/latest/streaming/apis-on-dataframes-and-datasets.html), Apache Spark documentation 11. [Change data feed](https://docs.delta.io/delta-change-data-feed/), Delta Lake documentation 12. [Constraints](https://docs.delta.io/delta-constraints/), Delta Lake documentation 13. [Manage data quality with pipeline expectations](https://docs.databricks.com/aws/en/ldp/expectations), Databricks documentation 14. [What happened to Delta Live Tables (DLT)?](https://docs.databricks.com/aws/en/ldp/where-is-dlt), Databricks documentation 15. [Add data tests to your DAG](https://docs.getdbt.com/docs/build/data-tests), dbt Developer Hub 16. [Apache Hudi overview](https://hudi.apache.org/docs/overview), Apache Hudi documentation 17. [Apache Iceberg tables](https://docs.snowflake.com/en/user-guide/tables-iceberg), Snowflake documentation 18. [Universal Format (UniForm)](https://docs.delta.io/delta-uniform/), Delta Lake documentation 19. [Unity Catalog: open, multimodal catalog for data and AI](https://docs.unitycatalog.io/), Unity Catalog documentation 20. [The Apache Software Foundation Graduates Two Open Source Projects from Incubator](https://news.apache.org/foundation/entry/the-apache-software-foundation-graduates-two-open-source-projects-from-incubator), The ASF Blog (March 5, 2026) 21. [Connecting to the Data Catalog using AWS Glue Iceberg REST endpoint](https://docs.aws.amazon.com/glue/latest/dg/connect-glu-iceberg-rest.html), AWS Glue documentation 22. [How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview), dbt Developer Hub 23. [Staging: Preparing our atomic building blocks](https://docs.getdbt.com/best-practices/how-we-structure/2-staging), dbt Developer Hub 24. [Table utility commands (VACUUM)](https://docs.delta.io/delta-utility/), Delta Lake documentation 25. [About OpenLineage](https://openlineage.io/docs/), OpenLineage documentation