Apache Iceberg: how the table format works and how to run it

How does Iceberg turn files into a table?

The table spec says the format "tracks individual data files in a table instead of directories". Table state sits in metadata files, and every change creates a new metadata file and replaces the old one with an atomic swap. Below that pointer, the table is a tree of files that are immutable once written:

Each write produces a snapshot, the state of the table at one time. A snapshot reuses as much of the previous snapshot's metadata tree as possible to avoid high write volumes. The same page adds that, by avoiding file listing and rename operations, "Iceberg tables are compatible with any object store".

For catalogs backed by a metastore or database, the spec describes the commit in four steps:

1. Create a new metadata file based on the current one.
2. Write it under a unique name, <V+1>-<random-uuid>.metadata.json.
3. Ask the catalog to swap the table's pointer from V to V+1, with a check-and-put.
4. If the swap fails, another writer already created V+1, so go back to step 1.

Writers are optimistic. The spec says the atomic swap "provides the basis for serializable isolation", and the reliability docs add that readers "always use a consistent snapshot of the table without holding a lock". Databricks says Iceberg creates an abstraction layer that "allows ACID transactions on your data in object storage".

When two commits start from the same version, only one succeeds, and in most cases the other can be retried on the new version. Whether a retry is safe depends on the operation: appends can always be applied, but a compaction must first verify that the files it replaces are still in the table. By default a commit is retried 4 times (commit.retry.num-retries) within a total timeout of 30 minutes.

What does Iceberg give you that Parquet files in a folder do not?

Schema evolution by column ID. Iceberg tracks each column by a unique ID, so add, drop, rename, widen and reorder are metadata changes and no data file is rewritten. The docs contrast this with formats that track columns by name, which can inadvertently un-delete a column if a name is reused.

Hidden partitioning. You declare a transform of a column, such as days(ordered_at), and Iceberg produces the partition values itself. In Hive tables the writer must supply them, and the wrong date format "produces silently incorrect results, not query failures". In Iceberg, consumers do not need to know how the table is partitioned.

Partition evolution. Queries do not reference partition values directly, so the layout can change on a live table. Old data keeps its old layout, new data uses the new one, and each layout is planned separately. In the lab, REPLACE PARTITION FIELD days(ordered_at) WITH hours(ordered_at) left the two existing data files in partition spec 0, and the next insert landed in spec 1.

Time travel, rollback and tags. Snapshots share the manifests and data files they have in common. Spark reads an older snapshot with VERSION AS OF (a snapshot ID, or a branch or tag name) or TIMESTAMP AS OF, and rollback_to_snapshot rolls a table back to a snapshot ID. Branches and tags are named references to snapshots, and their default retention is forever.

Row-level deletes. There are two types: a position delete marks a row by data file path and row position, and an equality delete marks rows by one or more column values, like id = 5. The spec calls versions 1, 2 and 3 "complete and adopted by the community":

A DELETE is copy-on-write by default (write.delete.mode). In the lab it committed an overwrite snapshot that added 2 data files and deleted 1. Merge-on-read (v2 and above) wrote delete files instead: one position delete file on a v2 table, and one deletion vector, a .puffin file, on a v3 table.

Which catalog should hold the tables?

The spec says the older file-system scheme, which relies on an atomic rename as on HDFS, is deprecated and will be removed in version 4, and that it is "unsafe in object stores and local file systems". The spec's other scheme keeps the pointer in a metastore or database and swaps it with a check-and-put, so for production tables we recommend a catalog service.

The REST catalog protocol exists because pluggable catalogs "started to cause some practical problems": catalogs had to be implemented in multiple languages, and commercial offerings found it hard to support many different ones. The answer was a common API defined with OpenAPI, "analogous to Hive's thrift protocol for HMS". The spec lists several reasons it matters, among them:

• one client implementation per engine works with any catalog;
• change-based commits allow server-side deconfliction and retries;
• it supports credential vending or remote signing.

A client of an Iceberg REST catalog needs at minimum a uri. Adapted from the docs' Spark example, without its OAuth2 settings (the address is a placeholder):

spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0 \
  --conf spark.sql.catalog.lake=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.lake.type=rest \
  --conf spark.sql.catalog.lake.uri=https://catalog-service/api/catalog \
  --conf spark.sql.catalog.lake.warehouse=my_warehouse

In the OpenAPI definition of the table update call, a 409 means the commit failed and the client may retry, while 500, 502 and 504 mean the commit state is unknown. Iceberg 1.12.0 changed the REST client to retry POST requests that carry an Idempotency-Key on 408, 500, 502, 503 and 504.

As of October 2026, the catalogs' own pages say:

Which engines can read and write Iceberg?

The Iceberg home page names Spark, Trino, Flink, Presto, Hive and Impala. Support for format versions and features differs by engine. As of October 2026, the page linked in each row says:

Format v3 and row-level deletes are the uneven part. Pick the format version that every engine reading the table supports, and test a delete on the least capable one before you commit to it.

What does Iceberg look like in Spark SQL?

Every statement and output below comes from the lab, in UTC, with invented orders. The catalog is a local Hadoop catalog named lake. Its vN.metadata.json naming is the spec's deprecated file-system scheme, so treat it as a lab convenience. The session enabled IcebergSparkSessionExtensions.

Create a table partitioned by a transform of its timestamp, then insert. The first INSERT wrote two orders, and five single-row inserts followed.

CREATE TABLE lake.shop.orders (
  order_id bigint, customer_id bigint, ordered_at timestamp, status string, total decimal(10,2)
) USING iceberg
PARTITIONED BY (days(ordered_at));

INSERT INTO lake.shop.orders VALUES
  (1, 101, TIMESTAMP '2026-10-01 09:05:00', 'paid', 18.50),
  (2, 102, TIMESTAMP '2026-10-01 09:40:00', 'paid', 42.00);

Those six commits left six snapshots and this layout (sizes in bytes, UUIDs shortened):

data/ordered_at_day=2026-10-01/00000-2-<uuid>-0-00001.parquet   1571   a data file (six, 1508 to 1571)
metadata/<uuid>-m0.avro                                         7628   a manifest (one per commit)
metadata/snap-<snapshot-id>-1-<uuid>.avro                       4828   a manifest list (one per snapshot)
metadata/v7.metadata.json                                       9020   the current table metadata
metadata/version-hint.text                                         1   the Hadoop catalog's pointer

Each 1.5 KB data file came with a manifest about five times its size, so at this scale the metadata outweighs the data. Now read the past, name a snapshot, and change the schema:

SELECT count(*) AS orders, sum(total) AS revenue
FROM lake.shop.orders VERSION AS OF 2731554061298857453;          -- 3 orders, 67.75

ALTER TABLE lake.shop.orders CREATE TAG `eod-2026-10-01`
  AS OF VERSION 6432746159754078936 RETAIN 30 DAYS;
SELECT count(*) FROM lake.shop.orders VERSION AS OF 'eod-2026-10-01';  -- 4

ALTER TABLE lake.shop.orders ADD COLUMN channel string;
ALTER TABLE lake.shop.orders RENAME COLUMN status TO order_status;
SELECT count(*) FROM lake.shop.orders.snapshots;                  -- still 6
SELECT * FROM lake.shop.orders VERSION AS OF 8350664985331374993; -- still shows `status`

The two ALTER statements created no snapshot, and the first snapshot still reads with its old column name.

Data lake or data warehouse: where does Iceberg fit?

A data lake keeps files in low-cost storage with a file API and applies a schema when data is read. A warehouse applies the schema when data is written. That is the framing of the 2021 CIDR lakehouse paper by Armbrust, Ghodsi, Xin and Zaharia, researchers at Databricks, UC Berkeley and Stanford. They argue for the lakehouse, and name Delta Lake and Apache Iceberg among systems that "provide transactional views of a data lake". They also warn: "Although a metadata layer adds management capabilities, it is not sufficient to achieve good SQL performance." So Iceberg adds management features to a lake, and the paper says speed needs other work, such as caching, indexes and data layout.

Iceberg, Delta Lake or Hudi?

The three formats side by side, as of October 2026:

Hudi's overview describes an open table format "purpose-built for high performance writes on incremental data pipelines". Its table types page adds that merge-on-read tables combine lightweight log files with the base file using periodic compaction.

Interoperability exists, with limits. UniForm generates Iceberg metadata asynchronously, so one copy of the data files serves clients of all formats. It is read-only from the Iceberg side, and the docs warn that an external writer "may destroy your Delta table and cause data loss". It does not work on tables with deletion vectors enabled. Apache XTable (incubating) "is NOT a new or separate format". It translates metadata between Hudi, Delta and Iceberg, and does not support Hudi and Iceberg merge-on-read tables or Delta deletion vectors.

Decide in this order:

1. Writers. List the engines that must write to the table and check each one's documented support for the format version.
2. Existing estate. Delta tables can reach Iceberg clients through UniForm, and XTable can translate Hudi tables, both within their limits.
3. Workload. Compare Hudi's incremental-pipeline focus with Iceberg's focus on huge analytic tables shared by many engines.

How do you keep an Iceberg table healthy?

Snapshots accumulate until they are expired. The maintenance docs list expiring snapshots, removing old metadata files and deleting orphan files as recommended, and compacting data files and rewriting manifests as optional. You run them with Spark procedures, Java actions, or Flink's TableMaintenance API in a Flink job. The lab ran each one:

SELECT partition, count(*) AS files, sum(file_size_in_bytes) AS bytes
FROM lake.shop.orders.files GROUP BY partition;                    -- 3 files, 4592 bytes; 4 files, 6310 bytes
-- 3 and 4 small files per partition, below the default min-input-files of 5
CALL lake.system.rewrite_data_files(table => 'shop.orders');       -- 0 files rewritten
CALL lake.system.rewrite_data_files(table => 'shop.orders',
  options => map('min-input-files', '2'));                         -- 7 files rewritten into 2

CALL lake.system.rewrite_manifests('shop.orders');                 -- 8 manifests became 1

CALL lake.system.expire_snapshots(table => 'shop.orders',
  older_than => TIMESTAMP '2026-10-10 05:41:26', retain_last => 1);
-- older_than was set an hour ahead of the clock (a lab shortcut). Snapshots: 9 to 2.
-- Deleted 4 data files, 12 manifests and 7 manifest lists.

CALL lake.system.remove_orphan_files(table => 'shop.orders', dry_run => true);
-- listed one stray file that had been backdated by five days; the real call deleted it

Compaction commits a new replace snapshot and leaves the old files for older snapshots, so the earlier VERSION AS OF query still worked afterwards. By default rewrite_data_files targets 512 MB files.

Expiry is what ends time travel. After it ran, VERSION AS OF an expired snapshot failed with Cannot find snapshot with ID …, while the 30-day tag still resolved. expire_snapshots defaults to snapshots older than 5 days and keeps the latest one, and by default branches and tags never expire.

Streaming writers need regular upkeep: a high commit rate produces data files, manifests and snapshots, so the Spark Structured Streaming docs recommend a trigger interval of 1 minute at the minimum, regular snapshot expiry, compaction, and manifest rewrites after "fast" appends. For how upstream changes are captured, see the guide to change data capture. Flink commits after a successful checkpoint, and its docs call elapsedSecondsSinceLastSuccessfulCommit "an ideal alerting metric" for failed or missing commits.

Requests are part of the storage bill: Amazon S3 charges for requests by type and quantity, and LIST requests for any storage class are charged at the same rate as S3 Standard PUT, COPY, and POST requests. The wider storage bill is a FinOps question.

When is Iceberg the wrong choice?

• A warehouse already covers the need. Computese's data platform page says a warehouse suits structured reporting at modest volume, and that many teams start with one and grow into open table formats. For data that outgrows a single database, see big data analytics.
• Nobody will run the upkeep. The docs present compaction, expiry and orphan removal as operations you run. Some managed services take them over: BigQuery's managed tables document automatic storage optimization, and Databricks says Unity Catalog manages snapshot expiration and file compaction on its managed Iceberg tables.

How do you adopt Iceberg without a big-bang migration?

1. Pick one table with a measurable problem: frequent schema changes, several engines reading the same files, or a need to reproduce a past figure.
2. Choose the catalog first: a catalog service, preferably one that speaks the REST protocol, not file-system tables. Stay on v2 (the default) unless every engine that reads the table supports v3.
3. Create the table with a partition transform such as days(ts), and set the format version and delete mode explicitly.
4. Load it from the engine you already run, read it from a second engine, and compare counts and sums. In the lab, DuckDB 1.5.6 read the Spark-written table with iceberg_scan('<warehouse>/shop/orders', allow_moved_paths = true) and no catalog, and returned 8 orders and a total of 202.50, the same as Spark's last query.
5. Schedule the upkeep before the first release: compaction, manifest rewrites, snapshot expiry with tags for what auditors need, orphan removal at the 3-day default, and an alert on the time since the last successful commit.
6. Move the next table. For Hive or Spark tables, the snapshot procedure makes a light-weight test copy, and migrate replaces the table and keeps the original as a backup by default. For a Delta table, snapshotDeltaLakeTable builds an Iceberg table from its transactions and leaves the original unchanged. The snapshot keeps using the original's data files, so the docs prohibit actions such as expire_snapshots on it, and the migration module supports Delta tables at minReaderVersion 1 and minWriterVersion 2.
7. Reconcile old and new outputs side by side before you retire the old table.

Computese's data platform service covers lakehouse tables in Apache Iceberg and Delta Lake, with medallion layers, partitioning and cost controls. A proof of concept for a bank's regulatory data on Cloudera and Iceberg reproduced a filed figure from its Iceberg snapshot with time travel, on synthetic data. Its lesson fits the lab above: dropping and recreating a table erases its snapshots, so write with insert-overwrite instead.