Publish / file · All operations

Write a Parquet file

A variant of Write a columnar file

Write rows to one or more Parquet files, optionally partitioned and compressed.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

arcform has parquet_export for one unpartitioned Parquet file.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache DataFusion (3)
Apache Hop (1)
AWS Glue (1)
Azure Data Factory (1)
Duckle (2)
  • Checkpointread 2026-09-25

    pass rows through and write a Parquet snapshot to a path

  • Parquetread 2026-09-25

    write Parquet with COPY, optional partitioning and compression

EasyMorph (1)
Pentaho Data Integration (1)
  • Parquet Outputread 2026-09-27

    Map fields and write rows to Parquet files

Polars (2)
Talend (1)
VisiData (1)

Background

Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval.[1] Parquet provides high performance compression and encoding schemes to handle complex data in bulk.[1] The open-source project to build Parquet began as a joint effort between Twitter and Cloudera, and Apache Parquet 1.0 was released in July 2013.[2]

When to use it

Parquet is supported in many programming languages and analytics tools, but not all implementations support the same features of the format.[1] A workflow that mixes several Parquet implementations needs thorough testing to ensure compatibility and performance across them.[1] In DuckDB, the PARQUET_VERSION option chooses between Parquet V1 and V2 when writing, and V1 is the default.[3] Before writing a file with newer format features, check that every program that will read it supports them. (Meridian's view) Apache Parquet supports limited schema evolution, with the ability to add new columns and merge schemas that do not conflict.[2]

DuckDB writes Parquet files with its COPY statement.[4] In DuckDB, writing a large number of partitions can be very expensive, because many files are created.[5] For DuckDB's partitioned writes, at least 100 MB of data per partition is generally recommended.[5] Partition by a column that queries filter on and that has few distinct values, never by an identifier. (Meridian's view)

The method

Parquet was designed as an improvement on the Trevni columnar storage format created by Doug Cutting, the creator of Hadoop.[2] Parquet is built with complex nested data structures in mind, and uses the record shredding and assembly algorithm described in Google's Dremel paper.[6][2] Since April 27, 2015, Apache Parquet has been a top-level Apache Software Foundation project.[2]

A Parquet file consists of one or more row groups, a row group contains exactly one column chunk per column, and column chunks contain one or more pages.[7] In Parquet, a page is conceptually an indivisible unit in terms of compression and encoding.[7] Parquet writes the file metadata after the data to allow single-pass writing, and readers first read that metadata to find the column chunks they need.[8] Parquet allows compression schemes to be specified on a per-column level.[6] Parquet applies dictionary encoding automatically for data with a small number of unique values.[2] Parquet implements a hybrid of bit packing and run-length encoding, switching to whichever produces the best compression.[2]

In DuckDB, Parquet files are written with Snappy compression by default, and the COMPRESSION option selects others, including gzip, zstd, brotli and lz4.[3] In DuckDB, the ROW_GROUP_SIZE option sets the target number of rows in each row group, with a default of 122880.[3]

Sources

  1. [1]Overview | Parquetdocumentation
  2. [2]Apache Parquet - Wikipediasecondary source
  3. [3]COPY Statement – DuckDBdocumentation
  4. [4]Reading and Writing Parquet Files – DuckDBdocumentation
  5. [5]Partitioned Writes – DuckDBdocumentation
  6. [6]Motivation | Parquetdocumentation
  7. [7]Concepts | Parquetdocumentation
  8. [8]File Format | Parquetdocumentation