Acquire / file · All operations

Read an Avro file

A variant of Read a columnar file

Read Avro container files or messages into rows, using the schema embedded in them or given separately.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache DataFusion (1)
  • Avroread 2026-09-27

    Read Avro files as a table

Apache Hop (1)
Apache NiFi (1)
  • AvroReaderread 2026-09-27

    Read Avro data into records, with an embedded or an external schema

Apache SeaTunnel (1)
  • Avro formatread 2026-09-27

    Read and write Kafka messages in Avro

AWS Glue (1)
  • Avro (read)read 2026-09-27

    Read Avro data from Amazon S3 and other supported connection types

Azure Data Factory (1)
Duckle (1)
  • Avroread 2026-09-25

    read Avro container files; the run path decodes in Rust, inspect uses read_avro

Pentaho Data Integration (2)
  • Avro Inputread 2026-09-27

    Decode binary or JSON Avro data from files or incoming fields into fields

  • Avro input (deprecated)read 2026-09-27

    Deprecated Avro reader, replaced by Avro Input

Polars (1)
  • read_avroread 2026-09-27

    Read an Avro file into a frame

Talend (2)
  • tAvroInputread 2026-09-27

    Read records from Avro files in a distributed file system

  • tAvroStreamInputread 2026-09-27

    Watch a directory and read Avro files as they are created

Yggdrasil Decision Forests (1)
  • avro: (read)read 2026-09-27

    Read Avro files by typed path; recommended for large datasets; no Avro writer is registered

Background

When Apache Avro data is stored in a file, its schema is stored with it, so that the file may be processed later by any program.[1] Apache Avro is a data serialization system with a compact, fast, binary data format and a container file to store persistent data.[1] Binary encoded Avro data does not include type information or field names, so a schema must always be used to read it correctly.[2]

When to use it

Avro calls the schema used to write the data the writer's schema, and the schema the application expects the reader's schema.[2] An Avro reader may be programmed to read data into a different schema from the writer's, for example when fields have been added or removed since the data was written.[2] In Avro schema resolution, a writer's field that the reader's schema lacks is ignored, and a reader's field missing from the writer's schema takes its default value, or signals an error if it has none.[2]

Avro's single-object encoding, used for example for records kept in an Apache Kafka topic, carries an 8-byte fingerprint of the schema rather than the schema itself.[2] Apache NiFi's AvroReader reads Avro data that contains its own schema, or takes the schema from elsewhere through its Schema Access Strategy property.[3] DuckDB's read_avro function has no support for providing a separate Avro schema file.[4]

The method

Avro was developed within Apache's Hadoop project, and uses JSON for defining data types and protocols.[5] Avro schemas are composed of primitive types, such as null, int, long and string, and complex types: record, enum, array, map, union and fixed.[5] An Avro object container file consists of a file header, which holds file metadata including the schema, followed by one or more data blocks.[5] Avro container files store objects in blocks that may be compressed, with synchronization markers between blocks to permit efficient splitting of files for MapReduce processing.[2] Each Avro block's binary data can be efficiently extracted or skipped without deserializing the contents.[2]

DuckDB's avro extension simplifies an Avro union of a type and the null type to just that type, so a union of int and null becomes a DuckDB INTEGER that is sometimes NULL.[4] DuckDB's avro extension turns the entries of a single top-level Avro record into top-level columns.[4] DuckDB cannot express the recursive type definitions that Avro has.[4]

Compared with the alternatives

Avro is similar to Thrift and Protocol Buffers, but does not require running a code-generation program when a schema changes.[5] Because the schema is present when Avro data is read, considerably less type information needs to be encoded with the data, giving a smaller serialization size.[1] When an Avro schema changes, both the old and new schemas are present when the data is processed, so differences may be resolved using field names rather than manually assigned field IDs.[1]

Sources

  1. [1]Apache Avro™ 1.12.0 Documentation | Apache Avrodocumentation
  2. [2]Specification | Apache Avrostandard
  3. [3]AvroReader - Apache NiFidocumentation
  4. [4]Avro Extension – DuckDBdocumentation
  5. [5]Apache Avro - Wikipediasecondary source