Read a lakehouse table

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

Delta Lake time travel queries an older snapshot of a table, for uses such as re-creating analyses or reports and fixing mistakes in data.^[4] In Delta Lake, a table read twice may return different results if its data is updated in between, and time travel can fix the data returned.^[4] Delta Lake can time travel to a previous version only while both the log and the data files for that version are retained.^[4] Delta Lake deletes data files only when VACUUM is run, and by default a file becomes a candidate for VACUUM 7 days after it was deleted from the table.^[4] Pin the version or snapshot in any query whose answer has to be reproduced later, and find out how long the table keeps its history before relying on it. (Meridian's view)

Delta Lake's reference implementation stores data files in directories named after partition values by default, but a file's actual partition values must be read from the transaction log.^[2] Read a lakehouse table through a reader for its format, rather than globbing its data files with a plain Parquet reader. (Meridian's view) In Apache Hudi, a Merge-on-Read table writes updates and deletes to log files, which are merged with the base files during query execution.^[5] A read-optimized query on a Hudi Merge-on-Read table reads purely columnar base files, and can miss recent data that a snapshot query sees.^[5]

DuckDB reads an individual Iceberg table directly from storage by pointing at its metadata, which needs no catalog and is read-only.^[6] DuckDB's iceberg_scan reads a historical Iceberg snapshot when passed snapshot_from_id or snapshot_from_timestamp, which are mutually exclusive.^[6]

The method

A Delta table has a single serial history of atomic versions, named using contiguous, monotonically increasing integers.^[2] Delta log entries are JSON files in the _delta_log directory at the root of the table, each holding the actions that turn the previous version into the next.^[2] Delta writers commit by adding a new entry to the log, which records the data files to logically add and remove.^[2]

Apache Iceberg keeps table state in metadata files, and every change creates a new metadata file that replaces the old one with an atomic swap.^[3] Iceberg readers use the snapshot that was current when they loaded the table metadata, and are not affected by changes until they refresh.^[3] In Iceberg, a snapshot's data files are tracked by manifest files, which hold a row for each data file with its partition data and metrics.^[3] Iceberg was started at Netflix by Ryan Blue and Dan Weeks, and its development began in 2017.^[7]

In a Hudi Copy-on-Write table, record updates or deletes create new base files and no log files are written, so each query reads only base files.^[5]

Compared with the alternatives

Iceberg addresses the performance and usability challenges of Apache Hive tables in large and demanding data lake environments.^[7]

Sources

1. [1]Apache Parquet - Wikipediasecondary source
2. [2]delta/PROTOCOL.md at master · delta-io/delta · GitHubstandard
3. [3]Spec - Apache Iceberg™standard
4. [4]Table batch reads and writes | Delta Lakedocumentation
5. [5]Table & Query Types | Apache Hudidocumentation
6. [6]Iceberg Extension – DuckDBdocumentation
7. [7]Apache Iceberg - Wikipediasecondary source