Write a lake table

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

Reach for a table format when several jobs write to a table that changes, or readers need a consistent view while it does; a plain folder of files is enough when one job writes data that never changes. (Meridian's view) In Delta Lake, readers can tail the Delta transaction log to determine what data has been added in a given period, which allows efficient streaming.^[2] In Delta Lake, data files that are no longer in the latest version of a table can be lazily deleted by the vacuum command after a user-specified retention period, 7 days by default.^[2] Expect storage to hold several versions of a changing table's files until old ones are cleaned up, and plan the clean-up as part of the job. (Meridian's view)

In DuckDB, an Iceberg table read directly from storage is read-only, and writing needs an attached Iceberg catalog.^[4] DuckDB's delta extension can create Delta tables and append to them.^[5]

The method

The Iceberg table format tracks individual data files rather than directories, so writers can create data files in place and add them to the table only in an explicit commit.^[6] Every change to an Iceberg table's state creates a new metadata file and replaces the old metadata with an atomic swap.^[6] An Iceberg snapshot represents the state of a table at some time, and its data files are tracked by manifest files holding a row for each data file with its partition data and metrics.^[6] In Iceberg, if the snapshot on which an update is based is no longer current, the writer must retry the update based on the new current version.^[6] Version 1 of the Iceberg spec defines how to manage large analytic tables using the immutable file formats Parquet, Avro and ORC.^[6] Version 2 of the Iceberg spec adds delete files, which encode rows deleted in existing data files so that individual rows can be deleted without rewriting the files.^[6] Iceberg was started at Netflix by Ryan Blue and Dan Weeks, development began in 2017, and the project was donated to the Apache Software Foundation in November 2018.^[1]

Delta Lake's transactions are implemented using multi-version concurrency control (MVCC), keeping multiple copies of the data rather than immediately replacing files whose records are updated or removed.^[2] A Delta Lake writer first optimistically writes new data files, then commits by adding a new entry to the log that records which data files to logically add and remove.^[2] A Delta table has a single serial history of atomic versions, named using contiguous, monotonically-increasing integers.^[2] In Delta Lake, directories named after partition values are only a convention, and the actual partition values for a file must be read from the transaction log.^[2]

Apache Hudi has two table types, COPY_ON_WRITE and MERGE_ON_READ.^[3]

Compared with the alternatives

Iceberg can plan queries by reading only the relevant manifest files rather than scanning all data files or directories, which avoids expensive operations such as directory listing.^[1] All metadata for a Delta table is stored alongside the data, which removes the need for a separate metastore to read it and lets static tables be copied or moved with standard filesystem tools.^[2]

Sources

1. [1]Apache Iceberg - Wikipediasecondary source
2. [2]delta/PROTOCOL.md at master · delta-io/delta · GitHubstandard
3. [3]Overview | Apache Hudidocumentation
4. [4]Iceberg Extension – DuckDBdocumentation
5. [5]Delta Extension – DuckDBdocumentation
6. [6]Spec - Apache Iceberg™standard