Publish / file · All operations
Write an Arrow file or stream
A variant of Write a columnar file
Write rows as an Arrow IPC file or record-batch stream.
Status
- In arcform
Not built yet
arcform does not do this yet. Its place on the roadmap
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
Apache Hop (1)
Data stream outputread 2026-09-27Write rows to an Arrow file, random-access file or Flight data stream
Polars (3)
DataFrame.write_ipcread 2026-09-27Write a frame to an Arrow IPC (Feather) file
DataFrame.write_ipc_streamread 2026-09-27Write a frame to an Arrow IPC record-batch stream
LazyFrame.sink_ipcread 2026-09-27Run a lazy query in streaming mode and write the result to Arrow IPC
Power Query M (1)
Arrow.ToBinaryread 2026-09-27Write the rows of a table as an Apache Arrow IPC stream
VisiData (2)
arrow (saver)read 2026-09-23Save as Arrow IPC file
arrows (saver)read 2026-09-23Save as Arrow IPC stream
Background
Apache Arrow's interprocess communication (IPC) format serialises data in record batches into binary payloads that can be rebuilt without copying memory.[1] In Apache Arrow, a record batch is an ordered collection of arrays of the same length, which may have different data types.[1] Arrow IPC has a stream format and a file format, which extends the stream with a footer that enables random access to any record batch.[1]
When to use it
Apache Arrow is designed as a complement to on-disk columnar formats such as Apache Parquet and Apache ORC, for processing data in memory.[2] Systems that use or support Arrow can transfer data between them at little-to-no cost, and need not implement custom connectors for every other system.[3] Arrow IPC streams are not typically stored as files, and when they are, the recommended file extension is .arrows.[1] The recommended extension for an Arrow IPC file is .arrow, and files in that format were sometimes called Feather V2 or named with the .feather extension.[1] Write an IPC file when a reader will jump to particular batches, and a stream when the batches are consumed in order as they arrive. (Meridian's view) Use Arrow to hand data between programs, and Parquet or ORC for data that is meant to be kept. (Meridian's view)
The method
The Arrow columnar format provides analytical performance and data locality guarantees in exchange for comparatively more expensive mutation operations.[1] In Arrow, the contiguous columnar layout enables vectorisation using Single Instruction, Multiple Data (SIMD) operations in modern processors.[3] Arrow uses Google's Flatbuffers project for metadata serialisation.[1]
An Arrow IPC stream is a sequence of encapsulated messages, in which the schema comes first and is the same for all the record batches that follow.[1] In an Arrow IPC stream, dictionary-encoded fields bring DictionaryBatch messages, and a dictionary key should be defined in a DictionaryBatch before a record batch uses it.[1] An Arrow stream writer can signal end-of-stream by writing a 4-byte continuation indicator followed by a zero metadata length, or by closing the stream interface.[1] An Arrow IPC file starts and ends with the magic string ARROW1, and its footer holds a redundant copy of the schema plus memory offsets and sizes for each data block.[1]
Apache Arrow was announced by the Apache Software Foundation on February 17, 2016, and its initial codebase and Java library were seeded by code from Apache Drill.[2]
Compared with the alternatives
The Arrow and Parquet projects include libraries that allow reading and writing data between the two formats.[2]
Sources
- [1]Arrow Columnar Format — Apache Arrow v26.0.1standard
- [2]Apache Arrow - Wikipediasecondary source
- [3]Format | Apache Arrowdocumentation