# Apache Iceberg: how the table format works and how to run it > Apache Iceberg is an open table format for analytic datasets. How commits, catalogs and engines work, Spark SQL tested on 1.12.0, and the upkeep a table needs. - URL: https://computese.com/apache-iceberg/ - Author: Duong Quan Nguyen, CEO, Computese - Published: 2026-11-08 - Updated: 2026-11-08 - Topics: Data ## In short - Iceberg tracks a table as a tree: a catalog pointer, a metadata file, a manifest list per snapshot, manifests and data files. A commit atomically swaps the pointer to a new metadata file, so readers always use a consistent snapshot. - Schema changes are metadata changes, and the partition layout can change on a live table while old data stays as written. Time travel reads an older snapshot, and it ends for any snapshot that expire_snapshots removes unless a tag or branch keeps it. - Format v2 added row-level deletes through delete files, and v3 added deletion vectors. As of October 2026 engines differ: Trino calls v3 experimental, Athena works on v2 tables only, and Snowflake and Databricks list v1 to v3 with exceptions. - A high commit rate produces more files, manifests and snapshots, so the docs recommend upkeep for streaming tables: compaction, manifest rewrites, snapshot expiry, and removal of old metadata and orphan files. Some managed services run part of it. - Delta Lake and Apache Hudi are the two other lakehouse table formats compared here. Delta UniForm lets Iceberg clients read Delta tables, and Apache XTable translates metadata between all three, each with documented limits. [Apache Iceberg](https://iceberg.apache.org/) is an open table format for huge analytic tables. It tracks a table's individual data files in metadata, so engines such as Spark, Trino and Flink can safely work with the same table at the same time. Each commit atomically swaps one metadata file for another, and the same metadata supports schema evolution and time travel. "The lab" below is a local run of Spark 4.0.4 with Iceberg 1.12.0, the [latest release](https://iceberg.apache.org/releases/) as of October 2026. For how to organize tables into layers and model them, see the guides on [medallion architecture](https://computese.com/medallion-architecture/) and [star schema](https://computese.com/star-schema/). ## How does Iceberg turn files into a table? The [table spec](https://iceberg.apache.org/spec/) says the format "tracks individual data files in a table instead of directories". Table state sits in metadata files, and every change creates a new metadata file and replaces the old one with an atomic swap. Below that pointer, the table is a tree of files that are immutable once written: | Layer | What it holds | | ------------- | ---------------------------------------------------------------------------------- | | Catalog | The location of the table's current metadata file | | Metadata file | JSON with the schema, partitioning, properties and snapshots | | Manifest list | One per snapshot: its manifests, with file counts and partition value summaries | | Manifest | An Avro file listing data files or delete files, with partition values and metrics | | Data file | The rows, in Parquet, Avro or ORC | Each write produces a snapshot, the state of the table at one time. A snapshot [reuses as much of the previous snapshot's metadata tree as possible](https://iceberg.apache.org/docs/latest/reliability/) to avoid high write volumes. The same page adds that, by avoiding file listing and rename operations, "Iceberg tables are compatible with any object store". ![A pointer box at the top moves from an older metadata document to a newer one; both documents reach a shared row of data-file blocks through list sheets and manifest sheets.](https://computese.com/images/blog/apache-iceberg/metadata-tree.80b4ea174a-1536.webp) *A commit writes a new metadata file and moves one pointer; everything unchanged below it is reused.* For catalogs backed by a metastore or database, the spec describes the commit in four steps: 1. Create a new metadata file based on the current one. 2. Write it under a unique name, `-.metadata.json`. 3. Ask the catalog to swap the table's pointer from V to V+1, with a check-and-put. 4. If the swap fails, another writer already created V+1, so go back to step 1. Writers are optimistic. The spec says the atomic swap "provides the basis for serializable isolation", and the reliability docs add that readers "always use a consistent snapshot of the table without holding a lock". [Databricks](https://docs.databricks.com/aws/en/iceberg/) says Iceberg creates an abstraction layer that "allows ACID transactions on your data in object storage". When two commits start from the same version, only one succeeds, and in most cases the other can be retried on the new version. Whether a retry is safe depends on the operation: appends can always be applied, but a compaction must first verify that the files it replaces are still in the table. By default a commit is [retried 4 times](https://iceberg.apache.org/docs/latest/configuration/) (`commit.retry.num-retries`) within a total timeout of 30 minutes. ![Two laptops send new documents toward one pointer box; the first passes through with a check mark and the second is sent back to its laptop by an orange looping arrow.](https://computese.com/images/blog/apache-iceberg/two-writers-one-pointer.e9343244ae-1536.webp) *Two writers can prepare a commit at once, but the pointer accepts one and the other usually retries.* ## What does Iceberg give you that Parquet files in a folder do not? **Schema evolution by column ID.** Iceberg tracks each column by a unique ID, so [add, drop, rename, widen and reorder](https://iceberg.apache.org/docs/latest/evolution/) are metadata changes and no data file is rewritten. The docs contrast this with formats that track columns by name, which can inadvertently un-delete a column if a name is reused. **Hidden partitioning.** You declare a transform of a column, such as `days(ordered_at)`, and Iceberg produces the partition values itself. In Hive tables [the writer must supply them](https://iceberg.apache.org/docs/latest/partitioning/), and the wrong date format "produces silently incorrect results, not query failures". In Iceberg, consumers do not need to know how the table is partitioned. **Partition evolution.** Queries do not reference partition values directly, so the layout can change on a live table. Old data keeps its old layout, new data uses the new one, and each layout is planned separately. In the lab, `REPLACE PARTITION FIELD days(ordered_at) WITH hours(ordered_at)` left the two existing data files in partition spec 0, and the next insert landed in spec 1. **Time travel, rollback and tags.** Snapshots share the manifests and data files they have in common. Spark reads an older snapshot with [`VERSION AS OF`](https://iceberg.apache.org/docs/latest/spark-queries/) (a snapshot ID, or a branch or tag name) or `TIMESTAMP AS OF`, and `rollback_to_snapshot` rolls a table back to a snapshot ID. Branches and tags are named references to snapshots, and [their default retention is forever](https://iceberg.apache.org/docs/latest/branching/). ![An older document links to four data-file blocks in a row of five, and a newer document links to all five; the fifth block is orange.](https://computese.com/images/blog/apache-iceberg/snapshots-share-files.a7299e5f11-1536.webp) *Snapshots share the data files they have in common instead of copying them.* **Row-level deletes.** There are two types: a position delete marks a row by data file path and row position, and an equality delete marks rows by one or more column values, like `id = 5`. The spec calls versions 1, 2 and 3 "complete and adopted by the community": | Format version | What the spec says it adds | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | v2 | Row-level updates and deletes through delete files; the default `format-version` since Iceberg 1.4.0 | | v3 | Deletion vectors, row lineage, variant, geometry and geography types, nanosecond timestamps, default values, multi-argument transforms, table encryption keys; position delete files "must not be added to v3 tables" | A `DELETE` is copy-on-write by default (`write.delete.mode`). In the lab it committed an `overwrite` snapshot that added 2 data files and deleted 1. Merge-on-read (v2 and above) wrote delete files instead: one position delete file on a v2 table, and one deletion vector, a `.puffin` file, on a v3 table. ## Which catalog should hold the tables? The spec says the older file-system scheme, which relies on an atomic rename as on HDFS, is [deprecated and will be removed in version 4](https://iceberg.apache.org/spec/), and that it is "unsafe in object stores and local file systems". The spec's other scheme keeps the pointer in a metastore or database and swaps it with a check-and-put, so for production tables we recommend a catalog service. The [REST catalog protocol](https://iceberg.apache.org/rest-catalog-spec/) exists because pluggable catalogs "started to cause some practical problems": catalogs had to be implemented in multiple languages, and commercial offerings found it hard to support many different ones. The answer was a common API defined with OpenAPI, "analogous to Hive's thrift protocol for HMS". The spec lists several reasons it matters, among them: - one client implementation per engine works with any catalog; - change-based commits allow server-side deconfliction and retries; - it supports credential vending or remote signing. A client of an [Iceberg REST catalog](https://iceberg.apache.org/docs/latest/rest-catalog/) needs at minimum a `uri`. Adapted from the docs' Spark example, without its OAuth2 settings (the address is a placeholder): ```bash spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0 \ --conf spark.sql.catalog.lake=org.apache.iceberg.spark.SparkCatalog \ --conf spark.sql.catalog.lake.type=rest \ --conf spark.sql.catalog.lake.uri=https://catalog-service/api/catalog \ --conf spark.sql.catalog.lake.warehouse=my_warehouse ``` In the [OpenAPI definition](https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml) of the table update call, a 409 means the commit failed and the client may retry, while 500, 502 and 504 mean the commit state is unknown. [Iceberg 1.12.0 changed the REST client](https://iceberg.apache.org/releases/) to retry POST requests that carry an Idempotency-Key on 408, 500, 502, 503 and 504. ![Three different engines connect to one central orange catalog box, which points to a metadata document resting on a storage cylinder full of data-file blocks.](https://computese.com/images/blog/apache-iceberg/rest-catalog-hub.7667e13e79-1536.webp) *Engines that share a catalog ask it which metadata file is current, so they see the same table.* As of October 2026, the catalogs' own pages say: | Catalog | What the page says | | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Apache Polaris](https://polaris.apache.org/) | An open-source, fully-featured catalog for Iceberg that implements Iceberg's REST API. The Apache Software Foundation [announced](https://news.apache.org/foundation/entry/the-apache-software-foundation-graduates-two-open-source-projects-from-incubator) it as a top-level project on March 5, 2026 | | [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/connect-glu-iceberg-rest.html) | An Iceberg REST endpoint, `https://glue..amazonaws.com/iceberg`, that supports table spec v1 and v2, defaulting to v2 | | [Project Nessie](https://projectnessie.org/) | A transactional catalog with Git-like semantics and cross-table transactions that works with Iceberg tables | | [Unity Catalog](https://docs.unitycatalog.io/) (open source) | Compatible with Iceberg's REST catalog API; its docs, last updated November 2025, call it "currently a sandbox project with LF AI and Data Foundation" | ## Which engines can read and write Iceberg? The Iceberg home page names Spark, Trino, Flink, Presto, Hive and Impala. Support for format versions and features differs by engine. As of October 2026, the page linked in each row says: | Engine | What its documentation says | | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Spark and Flink | In Iceberg's [multi-engine table](https://iceberg.apache.org/multi-engine-support/), Spark 3.5, 4.0 and 4.1 and Flink 1.20, 2.2 and 2.3 are "Maintained" | | [Trino](https://trino.io/docs/current/connector/iceberg.html) | Spec versions 1 and 2; "Support for format version 3 is experimental" | | [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) | Iceberg 1.4.2; "only creates and operates on Iceberg v2 tables"; only tables created against the AWS Glue catalog | | [Snowflake](https://docs.snowflake.com/en/user-guide/tables-iceberg) | Parquet only; versions 1, 2 and 3, "excluding" some features, such as row-level equality deletes | | [BigQuery](https://docs.cloud.google.com/bigquery/docs/biglake-iceberg-tables-in-bigquery) | "Apache Iceberg managed tables" with DML, Storage Write API streaming, time travel and automatic storage optimization | | [DuckDB](https://duckdb.org/docs/current/core_extensions/iceberg/overview.html) | Reading a table directly from storage "requires no catalog and is read-only"; attaching a REST catalog gives the full feature set, including writing | | [Databricks](https://docs.databricks.com/aws/en/iceberg/) | Parquet and versions 1, 2 and 3; for v2, position and equality deletes "aren't supported"; branching and tagging "aren't supported"; managed tables do not support partitioning by expressions such as `days()` or `bucket()` | Format v3 and row-level deletes are the uneven part. Pick the format version that every engine reading the table supports, and test a delete on the least capable one before you commit to it. ## What does Iceberg look like in Spark SQL? Every statement and output below comes from the lab, in UTC, with invented orders. The catalog is a local Hadoop catalog named `lake`. Its `vN.metadata.json` naming is the spec's deprecated file-system scheme, so treat it as a lab convenience. The session enabled `IcebergSparkSessionExtensions`. Create a table partitioned by a transform of its timestamp, then insert. The first `INSERT` wrote two orders, and five single-row inserts followed. ```sql CREATE TABLE lake.shop.orders ( order_id bigint, customer_id bigint, ordered_at timestamp, status string, total decimal(10,2) ) USING iceberg PARTITIONED BY (days(ordered_at)); INSERT INTO lake.shop.orders VALUES (1, 101, TIMESTAMP '2026-10-01 09:05:00', 'paid', 18.50), (2, 102, TIMESTAMP '2026-10-01 09:40:00', 'paid', 42.00); ``` Those six commits left six snapshots and this layout (sizes in bytes, UUIDs shortened): ```text data/ordered_at_day=2026-10-01/00000-2--0-00001.parquet 1571 a data file (six, 1508 to 1571) metadata/-m0.avro 7628 a manifest (one per commit) metadata/snap--1-.avro 4828 a manifest list (one per snapshot) metadata/v7.metadata.json 9020 the current table metadata metadata/version-hint.text 1 the Hadoop catalog's pointer ``` Each 1.5 KB data file came with a manifest about five times its size, so at this scale the metadata outweighs the data. Now read the past, name a snapshot, and change the schema: ```sql SELECT count(*) AS orders, sum(total) AS revenue FROM lake.shop.orders VERSION AS OF 2731554061298857453; -- 3 orders, 67.75 ALTER TABLE lake.shop.orders CREATE TAG `eod-2026-10-01` AS OF VERSION 6432746159754078936 RETAIN 30 DAYS; SELECT count(*) FROM lake.shop.orders VERSION AS OF 'eod-2026-10-01'; -- 4 ALTER TABLE lake.shop.orders ADD COLUMN channel string; ALTER TABLE lake.shop.orders RENAME COLUMN status TO order_status; SELECT count(*) FROM lake.shop.orders.snapshots; -- still 6 SELECT * FROM lake.shop.orders VERSION AS OF 8350664985331374993; -- still shows `status` ``` The two `ALTER` statements created no snapshot, and the first snapshot still reads with its old column name. ## Data lake or data warehouse: where does Iceberg fit? A data lake keeps files in low-cost storage with a file API and applies a schema when data is read. A warehouse applies the schema when data is written. That is the framing of the 2021 [CIDR lakehouse paper](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) by Armbrust, Ghodsi, Xin and Zaharia, researchers at Databricks, UC Berkeley and Stanford. They argue for the lakehouse, and name Delta Lake and Apache Iceberg among systems that "provide transactional views of a data lake". They also warn: "Although a metadata layer adds management capabilities, it is not sufficient to achieve good SQL performance." So Iceberg adds management features to a lake, and the paper says speed needs other work, such as caching, indexes and data layout. ## Iceberg, Delta Lake or Hudi? The three formats side by side, as of October 2026: | Aspect | Apache Iceberg | Delta Lake | Apache Hudi | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | Governance | [Apache Software Foundation committee](https://projects.apache.org/json/foundation/committees.json), established May 2020 | [Linux Foundation Projects](https://delta.io/) sub-project, joined in 2019 | Apache Software Foundation committee, established May 2020 | | Engines named on its site | Spark, Trino, Flink, Presto, Hive and Impala | Spark, PrestoDB, Flink, Trino, Hive, Snowflake, BigQuery, Athena, Redshift and Databricks, among others | Spark, Flink, Presto, Trino and Hive, among others | | Table state | Metadata files, manifest lists and manifests | A `_delta_log` of zero-padded JSON entries, plus checkpoints | A [timeline](https://hudi.apache.org/docs/timeline), "a log of all actions" | | A commit | An atomic swap of the metadata pointer | Writers "optimistically write out new data files", then add an entry to the [log](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) | An action on the timeline, such as COMMIT, DELTA_COMMIT or REPLACE_COMMIT | | Row-level changes | Delete files (v2) and deletion vectors (v3), copy-on-write or merge-on-read | Optional deletion vector files | [Copy-on-write and merge-on-read](https://hudi.apache.org/docs/table_types) tables | | Read as another format | XTable can translate it | UniForm lets Iceberg and Hudi clients read it; XTable can translate it | XTable can translate it | Hudi's [overview](https://hudi.apache.org/docs/overview) describes an open table format "purpose-built for high performance writes on incremental data pipelines". Its table types page adds that merge-on-read tables combine lightweight log files with the base file using periodic compaction. Interoperability exists, with limits. [UniForm](https://docs.delta.io/delta-uniform/) generates Iceberg metadata asynchronously, so one copy of the data files serves clients of all formats. It is read-only from the Iceberg side, and the docs warn that an external writer "may destroy your Delta table and cause data loss". It does not work on tables with deletion vectors enabled. [Apache XTable](https://xtable.apache.org/) (incubating) "is NOT a new or separate format". It translates metadata between Hudi, Delta and Iceberg, and does not support Hudi and Iceberg merge-on-read tables or Delta deletion vectors. Decide in this order: 1. **Writers.** List the engines that must write to the table and check each one's documented support for the format version. 2. **Existing estate.** Delta tables can reach Iceberg clients through UniForm, and XTable can translate Hudi tables, both within their limits. 3. **Workload.** Compare Hudi's incremental-pipeline focus with Iceberg's focus on huge analytic tables shared by many engines. ## How do you keep an Iceberg table healthy? Snapshots accumulate until they are expired. The [maintenance docs](https://iceberg.apache.org/docs/latest/maintenance/) list expiring snapshots, removing old metadata files and deleting orphan files as recommended, and compacting data files and rewriting manifests as optional. You run them with Spark procedures, Java actions, or Flink's [TableMaintenance API](https://iceberg.apache.org/docs/latest/flink-maintenance/) in a Flink job. The lab ran each one: ```sql SELECT partition, count(*) AS files, sum(file_size_in_bytes) AS bytes FROM lake.shop.orders.files GROUP BY partition; -- 3 files, 4592 bytes; 4 files, 6310 bytes -- 3 and 4 small files per partition, below the default min-input-files of 5 CALL lake.system.rewrite_data_files(table => 'shop.orders'); -- 0 files rewritten CALL lake.system.rewrite_data_files(table => 'shop.orders', options => map('min-input-files', '2')); -- 7 files rewritten into 2 CALL lake.system.rewrite_manifests('shop.orders'); -- 8 manifests became 1 CALL lake.system.expire_snapshots(table => 'shop.orders', older_than => TIMESTAMP '2026-10-10 05:41:26', retain_last => 1); -- older_than was set an hour ahead of the clock (a lab shortcut). Snapshots: 9 to 2. -- Deleted 4 data files, 12 manifests and 7 manifest lists. CALL lake.system.remove_orphan_files(table => 'shop.orders', dry_run => true); -- listed one stray file that had been backdated by five days; the real call deleted it ``` Compaction commits a new `replace` snapshot and leaves the old files for older snapshots, so the earlier `VERSION AS OF` query still worked afterwards. By default [`rewrite_data_files`](https://iceberg.apache.org/docs/latest/spark-procedures/) targets 512 MB files. ![A loose pile of tiny blocks passes through a gear and becomes one large orange block, while faint dashed copies of the tiny blocks stay linked to an older document.](https://computese.com/images/blog/apache-iceberg/small-files-compaction.569b41792d-1536.webp) *Compaction adds a snapshot and leaves the old files for older snapshots, so storage shrinks only after expiry.* Expiry is what ends time travel. After it ran, `VERSION AS OF` an expired snapshot failed with `Cannot find snapshot with ID …`, while the 30-day tag still resolved. `expire_snapshots` defaults to snapshots older than 5 days and keeps the latest one, and by default branches and tags never expire. ![A faded older document has its line cut by scissors and points only to two dashed blocks, the current document links to four blocks, and one orange block with no line to any document has a magnifying glass above it.](https://computese.com/images/blog/apache-iceberg/expire-and-orphans.70fcded9cb-1536.webp) *Expiry removes what no live snapshot needs; orphan removal finds files no snapshot ever referenced.* > [!WARNING] > `expire_snapshots` ends time travel for every snapshot it removes, so tag what an audit or a rollback plan needs first. The maintenance docs call it "dangerous" to remove orphan files with a retention interval shorter than the time expected for any write to complete, because in-progress files might be treated as orphans. The default interval is 3 days. Streaming writers need regular upkeep: a high commit rate produces data files, manifests and snapshots, so the [Spark Structured Streaming docs](https://iceberg.apache.org/docs/latest/spark-structured-streaming/) recommend a trigger interval of 1 minute at the minimum, regular snapshot expiry, compaction, and manifest rewrites after "fast" appends. For how upstream changes are captured, see the guide to [change data capture](https://computese.com/change-data-capture/). Flink commits after a successful checkpoint, and its [docs](https://iceberg.apache.org/docs/latest/flink-writes/) call `elapsedSecondsSinceLastSuccessfulCommit` "an ideal alerting metric" for failed or missing commits. Requests are part of the storage bill: Amazon S3 [charges for requests](https://aws.amazon.com/s3/pricing/) by type and quantity, and LIST requests for any storage class are charged at the same rate as S3 Standard PUT, COPY, and POST requests. The wider storage bill is a FinOps question. | Symptom | Likely cause | Fix | | --------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | Queries open many tiny files | Many small commits, often from streaming | `rewrite_data_files`, and a trigger interval of 1 minute or more | | Storage does not shrink after deletes or compaction | Older snapshots still reference the old files | `expire_snapshots`; data files go once no snapshot needs them | | `VERSION AS OF` fails with "Cannot find snapshot" | The snapshot was expired | Tag snapshots you must keep, with `RETAIN n DAYS`, before expiry runs | | Unreferenced files pile up | Task or job failures leave files no metadata references | `remove_orphan_files`, `dry_run` first, `older_than` longer than your slowest write (default 3 days) | | Many `vN.metadata.json` files | Old metadata files are kept by default | `write.metadata.delete-after-commit.enabled=true`; `write.metadata.previous-versions-max` (default 100) sets how many are tracked | | Table history vanished | `DROP TABLE` then `CREATE TABLE` left 0 snapshots in the lab | `CREATE OR REPLACE TABLE`, which kept history in the lab | ## When is Iceberg the wrong choice? - **A warehouse already covers the need.** Computese's data platform page says a warehouse suits structured reporting at modest volume, and that many teams start with one and grow into open table formats. For data that outgrows a single database, see [big data analytics](https://computese.com/the-power-of-big-data-analytics/). - **Nobody will run the upkeep.** The docs present compaction, expiry and orphan removal as operations you run. Some managed services take them over: BigQuery's managed tables document automatic storage optimization, and Databricks says Unity Catalog manages snapshot expiration and file compaction on its managed Iceberg tables. ## How do you adopt Iceberg without a big-bang migration? 1. **Pick one table with a measurable problem:** frequent schema changes, several engines reading the same files, or a need to reproduce a past figure. 2. **Choose the catalog first:** a catalog service, preferably one that speaks the REST protocol, not file-system tables. Stay on v2 (the default) unless every engine that reads the table supports v3. 3. **Create the table with a partition transform** such as `days(ts)`, and set the format version and delete mode explicitly. 4. **Load it from the engine you already run,** read it from a second engine, and compare counts and sums. In the lab, DuckDB 1.5.6 read the Spark-written table with `iceberg_scan('/shop/orders', allow_moved_paths = true)` and no catalog, and returned 8 orders and a total of 202.50, the same as Spark's last query. 5. **Schedule the upkeep before the first release:** compaction, manifest rewrites, snapshot expiry with tags for what auditors need, orphan removal at the 3-day default, and an alert on the time since the last successful commit. 6. **Move the next table.** For Hive or Spark tables, the [`snapshot` procedure](https://iceberg.apache.org/docs/latest/spark-procedures/) makes a light-weight test copy, and `migrate` replaces the table and keeps the original as a backup by default. For a Delta table, `snapshotDeltaLakeTable` builds an Iceberg table from its transactions and leaves the original unchanged. The snapshot keeps using the original's data files, so the docs prohibit actions such as `expire_snapshots` on it, and the [migration module](https://iceberg.apache.org/docs/latest/delta-lake-migration/) supports Delta tables at minReaderVersion 1 and minWriterVersion 2. 7. **Reconcile old and new outputs side by side** before you retire the old table. Computese's [data platform service](https://computese.com/services/data-platform/) covers lakehouse tables in Apache Iceberg and Delta Lake, with medallion layers, partitioning and cost controls. A [proof of concept for a bank's regulatory data on Cloudera and Iceberg](https://computese.com/work/banking-regulatory-data/) reproduced a filed figure from its Iceberg snapshot with time travel, on synthetic data. Its lesson fits the lab above: dropping and recreating a table erases its snapshots, so write with insert-overwrite instead. ## Key terms - **Open table format**: A published specification for how a table's schema, partitions, snapshots and data files are tracked in metadata, so that several engines can work on the same table. - **Catalog**: The component that holds the location of a table's current metadata file and swaps it atomically on each commit. - **Snapshot**: The state of a table at one point in time. Each write produces one, and its manifests are stored in a manifest list file. - **Manifest**: An immutable Avro file that lists data files or delete files with each file's partition values and column metrics. Snapshots reuse manifests. - **Hidden partitioning**: Partition values that Iceberg produces by transforming a column, such as days(ts), so queries filter on the column and do not depend on the table's physical layout. - **Deletion vector**: A bitmap that identifies the deleted rows within a single data file by position, stored as a Puffin deletion-vector-v1 blob. Added in format v3, where position delete files must not be added. - **Compaction**: Rewriting many small data files into fewer large ones. In Iceberg's Spark procedures it is rewrite_data_files, which reduces metadata overhead and file open cost. - **Orphan file**: A file under the table location that no table metadata references, for example one left by a failed task or job. The remove_orphan_files procedure deletes such files older than an interval, 3 days by default. - **Optimistic concurrency**: Writers assume the table will not change before their commit and prepare a new metadata file. If another writer committed first, the swap fails and, in most cases, the writer can retry on the new version. ## Common questions ### What is Apache Iceberg used for? Iceberg manages large analytic tables so that engines such as Spark, Trino and Flink can work with the same tables at the same time. The spec describes it as managing a large, slow-changing collection of files in a distributed file system or key-value store as a table. It adds schema evolution, hidden partitioning and time travel for tables of Parquet, Avro or ORC files. Its home page calls it the open table format for analytic datasets. ### Is Apache Iceberg a database or a file format? It is a table format, not a database or a data file format. The spec covers the metadata that tracks a table's files, which stay in formats such as Parquet, Avro or ORC. A catalog holds the pointer to the current metadata, and an engine such as Spark or Trino runs the queries. ### What is the difference between Delta Lake and Iceberg? Both add a metadata layer over data files in storage. Iceberg keeps metadata files, manifest lists and manifests, and a catalog swaps a pointer on each commit. Delta Lake appends entries to a transaction log in a _delta_log directory. Iceberg is an Apache Software Foundation project, and Delta Lake describes itself as a sub-project of the Linux Foundation Projects. ### What is an Iceberg REST catalog? It is any catalog service that implements the Iceberg REST Catalog API, so an engine needs only one client implementation to work with any compliant server. The protocol also allows server-side conflict handling and retries for commits, and supports credential vending or remote signing. Apache Polaris and the AWS Glue Iceberg REST endpoint are examples. ### Does Iceberg time travel keep history forever? No. Each write adds a snapshot, and expiring snapshots removes them from metadata, so they are no longer available for time travel. Spark's expire_snapshots procedure expires snapshots older than five days by default. By default, branches and tags never expire, so tagging a snapshot keeps it. ### What is Iceberg compaction and when should it run? Compaction rewrites small data files into larger ones, with the rewrite_data_files procedure in Spark, to reduce metadata overhead and file open cost. By default the target size is 512 MB, and a file group with five or more files is rewritten. The docs suggest using the files metadata table to inspect data file sizes and decide when to compact partitions. ### Is a data lake the same as a data warehouse? No. In the terms of the 2021 CIDR lakehouse paper, a data lake is low-cost storage with a file API and schema-on-read, while a warehouse writes data with schema-on-write. Table formats such as Iceberg add transactional views to a lake, but the paper notes that a metadata layer alone is not sufficient for good SQL performance. ## Sources 1. [Apache Iceberg: the open table format for analytic datasets](https://iceberg.apache.org/), Apache Iceberg 2. [Apache Iceberg releases](https://iceberg.apache.org/releases/), Apache Iceberg 3. [Iceberg Table Spec](https://iceberg.apache.org/spec/), Apache Iceberg 4. [Reliability](https://iceberg.apache.org/docs/latest/reliability/), Apache Iceberg documentation 5. [What is Apache Iceberg in Databricks?](https://docs.databricks.com/aws/en/iceberg/), Databricks documentation 6. [Configuration](https://iceberg.apache.org/docs/latest/configuration/), Apache Iceberg documentation 7. [Evolution](https://iceberg.apache.org/docs/latest/evolution/), Apache Iceberg documentation 8. [Partitioning](https://iceberg.apache.org/docs/latest/partitioning/), Apache Iceberg documentation 9. [Spark Queries](https://iceberg.apache.org/docs/latest/spark-queries/), Apache Iceberg documentation 10. [Branching and Tagging](https://iceberg.apache.org/docs/latest/branching/), Apache Iceberg documentation 11. [REST Catalog Spec](https://iceberg.apache.org/rest-catalog-spec/), Apache Iceberg 12. [Iceberg REST Catalog](https://iceberg.apache.org/docs/latest/rest-catalog/), Apache Iceberg documentation 13. [Iceberg REST catalog OpenAPI specification (main branch)](https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml), Apache Iceberg on GitHub 14. [Apache Polaris](https://polaris.apache.org/), Apache Polaris 15. [The Apache Software Foundation Graduates Two Open Source Projects from Incubator](https://news.apache.org/foundation/entry/the-apache-software-foundation-graduates-two-open-source-projects-from-incubator), The Apache Software Foundation 16. [Connecting to the Data Catalog using AWS Glue Iceberg REST endpoint](https://docs.aws.amazon.com/glue/latest/dg/connect-glu-iceberg-rest.html), Amazon Web Services 17. [Project Nessie: Transactional Catalog for Data Lakes with Git-like semantics](https://projectnessie.org/), Project Nessie 18. [Unity Catalog](https://docs.unitycatalog.io/), Unity Catalog documentation 19. [Multi-engine support](https://iceberg.apache.org/multi-engine-support/), Apache Iceberg 20. [Iceberg connector](https://trino.io/docs/current/connector/iceberg.html), Trino documentation 21. [Query Apache Iceberg tables](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html), Amazon Athena documentation 22. [Apache Iceberg tables](https://docs.snowflake.com/en/user-guide/tables-iceberg), Snowflake documentation 23. [Apache Iceberg managed tables](https://docs.cloud.google.com/bigquery/docs/biglake-iceberg-tables-in-bigquery), Google Cloud documentation 24. [Iceberg extension overview](https://duckdb.org/docs/current/core_extensions/iceberg/overview.html), DuckDB documentation 25. [Spark Procedures](https://iceberg.apache.org/docs/latest/spark-procedures/), Apache Iceberg documentation 26. [Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf), CIDR 2021 (Armbrust, Ghodsi, Xin, Zaharia) 27. [Apache committees (project data)](https://projects.apache.org/json/foundation/committees.json), The Apache Software Foundation 28. [Delta Lake](https://delta.io/), Delta Lake Project (Linux Foundation) 29. [Hudi timeline](https://hudi.apache.org/docs/timeline), Apache Hudi 30. [Delta Transaction Log Protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md), Delta Lake on GitHub 31. [Hudi table types](https://hudi.apache.org/docs/table_types), Apache Hudi 32. [Apache Hudi overview](https://hudi.apache.org/docs/overview), Apache Hudi 33. [Universal Format (UniForm)](https://docs.delta.io/delta-uniform/), Delta Lake documentation 34. [Apache XTable (Incubating)](https://xtable.apache.org/), Apache XTable 35. [Delta Lake Migration](https://iceberg.apache.org/docs/latest/delta-lake-migration/), Apache Iceberg documentation 36. [Maintenance](https://iceberg.apache.org/docs/latest/maintenance/), Apache Iceberg documentation 37. [Flink Table Maintenance](https://iceberg.apache.org/docs/latest/flink-maintenance/), Apache Iceberg documentation 38. [Spark Structured Streaming](https://iceberg.apache.org/docs/latest/spark-structured-streaming/), Apache Iceberg documentation 39. [Flink Writes](https://iceberg.apache.org/docs/latest/flink-writes/), Apache Iceberg documentation 40. [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/), Amazon Web Services