Publish / file · All operations

Write an Avro file

A variant of Write a columnar file

Write rows as Avro records to a container file.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache Hop (1)
Apache NiFi (1)
AWS Glue (1)
  • Avro (write)read 2026-09-27

    Write Avro files and folders to Amazon S3

Azure Data Factory (1)
Duckle (1)
  • Avroread 2026-09-25

    write an Avro container file

Pentaho Data Integration (1)
  • Avro Outputread 2026-09-27

    Serialise rows as binary or JSON Avro and write them to a file

Polars (1)
Talend (1)
  • tAvroOutputread 2026-09-27

    Write rows to Avro files in a distributed file system

Background

Apache Avro is a remote procedure call and data serialisation framework developed within Apache's Hadoop project.[1] An Avro object container file has a schema, and all objects stored in the file must be written to that schema, using binary encoding.[2] Avro data may be stored with its schema, so a serialised item can be read without knowing the schema ahead of time.[1]

When to use it

Avro schemas are defined using JavaScript Object Notation (JSON), and are composed of primitive types such as null, boolean, int, long, float, double, bytes and string, and complex types such as record, enum, array, map, union and fixed.[1][3] Avro's primary use is in Apache Hadoop, as a serialisation format for persistent data and as a wire format between Hadoop nodes and from client programs to Hadoop services.[1] Avro calls the schema used to write data the writer's schema and the schema an application expects the reader's schema, and the two may differ, for example when fields have been added to or removed from records.[2] In Avro schema resolution, the fields of two record schemas are matched by name, so their order may differ.[2] In Avro schema resolution, an int in the writer's schema may be promoted to a long, float or double in the reader's schema.[2] Reach for Avro when records pass between programs whose schemas change over time, and treat each schema change as something every reader must be able to resolve. (Meridian's view)

The method

An Avro container file header consists of four bytes, ASCII 'O', 'b', 'j' followed by 1, then the file metadata including the schema, then a 16-byte, randomly-generated sync marker.[2] In an Avro container file, objects are stored in blocks that may be compressed, and sync markers between blocks permit efficient splitting of files for MapReduce processing.[2] Each Avro data block holds a count of its objects, the size in bytes of the serialised objects, the objects themselves and the file's 16-byte sync marker.[2] Each Avro block's binary data can be efficiently extracted or skipped without deserialising the contents.[2] Avro implementations are required to support the null and deflate codecs, and a file with no codec is assumed to use null, which passes data through uncompressed.[2] For data blocks Avro specifies a binary and a JSON encoding, and most applications use the binary encoding because it is smaller and faster.[1]

Compared with the alternatives

Avro is similar to Thrift and Protocol Buffers, but does not require running a code-generation program when a schema changes, unless desired for statically-typed languages.[1]

Sources

  1. [1]Apache Avro - Wikipediasecondary source
  2. [2]Specification | Apache Avrostandard
  3. [3]JSON - Wikipediasecondary source