Acquire / file · All operations

Read an Arrow file or stream

A variant of Read a columnar file

Read Arrow IPC files, record-batch streams or Arrow datasets into a table.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache DataFusion (1)
  • Arrowread 2026-09-27

    Read Arrow IPC files as a table

Apache Hop (1)
  • Data stream inputread 2026-09-27

    Read rows from an Arrow file, random-access file or Flight data stream

DuckDB (2)
  • arrow_scanread 2026-09-27

    arrow_scan table function

  • arrow_scan_dumbread 2026-09-27

    arrow_scan_dumb table function

Polars (4)
  • read_ipcread 2026-09-27

    Read an Arrow IPC (Feather v2) file into a frame

  • read_ipc_streamread 2026-09-27

    Read an Arrow IPC record-batch stream into a frame

  • scan_ipcread 2026-09-27

    Lazily read one or many Arrow IPC files, by glob pattern

  • scan_pyarrow_datasetread 2026-09-27

    Lazily read a pyarrow dataset

Power Query M (2)
  • Arrow.Documentread 2026-09-27

    Read an Apache Arrow document into a table

  • Arrow.FromBinaryread 2026-09-27

    Read an Apache Arrow IPC stream, such as an HTTP response body, into a table

VisiData (2)

Background

Reading an Apache Arrow inter-process communication (IPC) file involves no decoding, because its on-disk representation is the same as the in-memory one.[1] Arrow IPC data can be passed between processes in the Arrow stream format, or persisted on storage first in the Arrow file format.[1] Analytics on an Arrow IPC file can use memory-mapping, avoiding any deserialization cost and extra copies.[1]

When to use it

In Apache Arrow, the primitive unit of serialized data is the record batch, an ordered collection of arrays of the same length.[2] In the Arrow IPC streaming format, the schema comes first in the stream and is the same for all of the record batches that follow.[2] The Arrow IPC file format supports random access to any record batch, through a footer holding a copy of the schema and the offsets and sizes of the data blocks.[2] Arrow IPC streams are not typically stored as files; when they are, the recommended extension is .arrows, while .arrow is recommended for the IPC file format.[2][1]

Feather version 2 is exactly the Arrow IPC file format, with the Feather name and APIs kept for backwards compatibility.[1] Feather version 1 files are distinct from Arrow IPC files and lack many features, such as the ability to store all Arrow data types.[3] The Arrow on-disk format is stable and will be readable by future versions of the libraries, but it does not prioritise the requirements of long-term archival storage.[1] Use Arrow files to hand data between tools and between steps of a pipeline, and keep data that has to be read years from now in a format built for archiving. (Meridian's view)

DuckDB reads the Arrow IPC format through its community nanoarrow extension, both from files and directly from stream buffers.[4] In DuckDB's nanoarrow extension, reading buffers directly is dangerous, because an incorrect pointer can crash the database system.[4]

The method

The Arrow columnar format includes a language-agnostic in-memory data structure specification, metadata serialization, and a protocol for serialization and generic data transport.[2] Arrow uses Google's Flatbuffers project for metadata serialization.[2] An Arrow IPC file starts and ends with the magic string ARROW1, and what lies between is identical to the stream format.[2] Arrow record batch body buffers may be uncompressed, or compressed with the lz4 or zstd codec.[2] Some Arrow implementations lack support for producing and consuming IPC data with compressed buffers.[2] The Arrow IPC format is little-endian by default, and some reader implementations may refuse to read IPC data with non-native endianness.[2]

The Apache Software Foundation announced Apache Arrow on February 17, 2016, and its initial codebase was seeded by code from Apache Drill.[5]

Compared with the alternatives

Apache Parquet files are often much smaller than Arrow IPC files, because of the columnar compression strategies Parquet uses.[1] Where disk storage or the network is slow, Parquet may be a better choice than Arrow IPC even for short-term storage or caching.[1] Apache Arrow and Apache Parquet complement each other and are commonly used together, with data stored on disk as Parquet and read into memory as Arrow.[1]

Sources

  1. [1]FAQ | Apache Arrowdocumentation
  2. [2]Arrow Columnar Format — Apache Arrow v26.0.1standard
  3. [3]Feather File Format — Apache Arrow v26.0.1documentation
  4. [4]nanoarrow – DuckDB Community Extensionsdocumentation
  5. [5]Apache Arrow - Wikipediasecondary source