Acquire / file · All operations

Read a Parquet file

A variant of Read a columnar file

Read one or many Parquet files into a table, reading only the columns and row groups the query needs.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

The DuckDB step

Read many Parquet files with only the columns and row groups a query needs

SQL
-- two files of two row groups each; the amounts rise with the order number, so each row group covers a range of amounts
COPY (SELECT i AS order_id, 'c' || (i % 7) AS customer, i::DECIMAL(8, 2) AS amount, repeat('x', 20) AS pad FROM range(1, 4097) t(i))
  TO '<your_scratch_dir>/pq_a.parquet' (FORMAT parquet, ROW_GROUP_SIZE 2048);
COPY (SELECT i AS order_id, 'c' || (i % 7) AS customer, i::DECIMAL(8, 2) AS amount, repeat('x', 20) AS pad FROM range(4097, 8193) t(i))
  TO '<your_scratch_dir>/pq_b.parquet' (FORMAT parquet, ROW_GROUP_SIZE 2048);
-- two files by one glob, two columns out of four, and a filter the row-group statistics can answer
CREATE TEMP TABLE result AS
SELECT customer, amount FROM read_parquet('<your_scratch_dir>/pq_*.parquet') WHERE amount >= 6000;
SELECT count(*) AS n, sum(amount) AS total FROM result;
-- the row groups the filter can rule out without reading them, from the file's own statistics
CREATE TEMP TABLE groups AS
SELECT file_name, row_group_id, max(stats_max_value::DOUBLE) AS max_amount
FROM parquet_metadata('<your_scratch_dir>/pq_*.parquet') WHERE path_in_schema = 'amount' GROUP BY ALL;
SELECT * FROM groups ORDER BY file_name, row_group_id;
SELECT CASE WHEN ((SELECT list(column_name) FROM (DESCRIBE result)) = ['customer', 'amount'] AND (SELECT count(*) = 2193 AND sum(amount) = 15561528 FROM result) AND (SELECT count(*) FROM read_parquet('<your_scratch_dir>/pq_*.parquet')) = 8192 AND (SELECT count(DISTINCT filename) FROM read_parquet('<your_scratch_dir>/pq_*.parquet', filename = true)) = 2 AND (SELECT count(*) = 4 AND count(*) FILTER (WHERE max_amount < 6000) = 2 FROM groups)) THEN 'as expected' ELSE error('the glob did not read 8192 rows from 2 files, the filter did not keep the 2193 orders of 6000 or more, or two of the four row groups were not ruled out by their statistics') END AS verdict;

Replace <your_scratch_dir> with a directory of your own that you can write to before running the statement.

  • The statement writes pq_a.parquet and pq_b.parquet itself, in <your_scratch_dir>.
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the glob did not read 8192 rows from 2 files, the filter did not keep the 2193 orders of 6000 or more, or two of the four row groups were not ruled out by their statistics”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache DataFusion (1)
  • Parquetread 2026-09-27

    Read Parquet files as a table, including nested types

Apache Hop (1)
AWS Glue (1)
  • Parquet (read)read 2026-09-27

    Read Parquet data from Amazon S3 and other supported connection types

Azure Data Factory (1)
DuckDB (2)
  • parquet_scanread 2026-09-27

    parquet_scan table function

  • read_parquetread 2026-09-27

    read_parquet table function

Duckle (1)
  • Parquetread 2026-09-25

    read Parquet files with read_parquet

EasyMorph (1)
KNIME Analytics Platform (1)
Pentaho Data Integration (1)
  • Parquet Inputread 2026-09-27

    Decode Parquet files and extract fields

Polars (2)
  • read_parquetread 2026-09-27

    Read a Parquet file into a frame

  • scan_parquetread 2026-09-27

    Lazily read local or cloud-hosted Parquet files

Power Query M (1)
  • Parquet.Documentread 2026-09-27

    Read a Parquet file binary into a table; named on the Parquet connector page only

Talend (2)
VisiData (1)

Background

Queries on Apache Parquet data that fetch specific column values need not read the entire row, which improves performance.[1] Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval.[2] A Parquet file splits a table into row groups, and its file metadata holds the start locations of every column chunk.[3]

When to use it

A Parquet row group is a partition of rows, consisting of a column chunk for each column in the dataset.[4] DuckDB pushes projections into a Parquet scan, so only the columns required for a query are read.[5] DuckDB writes per-column statistics, including minimum and maximum values, for every Parquet row group, and its reader uses them to skip row groups that cannot match a query's WHERE clause.[4] In DuckDB, sorting data on the columns filtered most often before writing a Parquet file makes each row group's minimum and maximum ranges tight and non-overlapping, so highly selective queries scan far fewer row groups.[4] When a Parquet file will be read again and again with the same filter, ask for it to be written sorted on that column. (Meridian's view)

DuckDB can only parallelise over row groups, so a Parquet file with a single giant row group can only be processed by a single thread.[6] DuckDB's parquet_metadata function shows a Parquet file's row groups and the statistics of its columns, which helps in working out what kind of skipping is possible.[7]

By default, DuckDB reads several Parquet files with the schema of the first file, unless the union_by_name option is set to unify their columns by name.[5] With union_by_name, DuckDB fills in NULL values for the columns a file does not have.[4] Parquet files from legacy writers may not set the UTF8 flag for strings, so DuckDB loads those columns as BLOB unless its binary_as_string option is set.[5] Not all Parquet implementations support the same features of the format, so a workflow that mixes several of them needs thorough testing for compatibility.[2] Before reading a folder of Parquet files with one glob, check that they share a schema, or name union_by_name so that a column missing from one file arrives as NULL rather than as a surprise. (Meridian's view)

The method

Apache Parquet began as a joint effort between Twitter and Cloudera, using the record shredding and assembly algorithm described in Google's Dremel.[1] The first version, Apache Parquet 1.0, was released in July 2013.[1]

Parquet file metadata is written after the data, to allow single-pass writing.[3] Parquet readers are expected to first read the file metadata to find the column chunks they are interested in, and then read those chunks sequentially.[3] In Parquet, compression is performed column by column, which lets different encoding schemes be used for text and integer data.[1] Parquet enables dictionary encoding dynamically for data with a small number of unique values.[1]

Compared with the alternatives

The DuckDB database format has HyperLogLog statistics that Parquet files do not have, and they improve the accuracy of cardinality estimates for queries with many joins.[6] Loading data from Parquet files into DuckDB needs approximately the same amount of space for the DuckDB database file.[6] Query Parquet in place for a one-off look or when disk is short, and load it into a database when the same data will be joined and queried many times. (Meridian's view)

Sources

  1. [1]Apache Parquet - Wikipediasecondary source
  2. [2]Overview | Parquetdocumentation
  3. [3]File Format | Parquetstandard
  4. [4]Parquet Tips – DuckDBdocumentation
  5. [5]Reading and Writing Parquet Files – DuckDBdocumentation
  6. [6]File Formats – DuckDBdocumentation
  7. [7]Querying Parquet Metadata – DuckDBdocumentation