Publish / objectstore · All operations

Upload to cloud storage

Put to S3, GCS, Azure Blob, HDFS, FTP, SFTP.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

Shape of the result
the table sent out, with nothing new to look at
Level
needs a DuckDB extension

Variants

Write metadata to stored objects

Set custom metadata on objects in a store or catalogue, leaving their contents alone.

Write the result to cloud storage

Write the pipeline's rows as a file in an object store or on a remote file server, in a chosen format.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Alteryx Designer (1)
Apache Hop (3)
Apache NiFi (1)
AWS Glue (1)
Azure Data Factory (1)
Duckle (5)
OpenRefine (1)
Pentaho Data Integration (3)
  • Catalog Outputread 2026-09-27

    Create or update a Pentaho Data Catalog resource and save its metadata

  • SFTP Putread 2026-09-27

    Upload a file or stream to a remote host over SFTP

  • Write metadata to HCP objectsread 2026-09-27

    Write custom metadata fields to a Hitachi Content Platform object

RapidMiner Studio (2)
Tableau Prep (1)
  • Amazon S3read 2026-09-27

    Save the result as a .parquet or .csv object in Amazon S3

Talend (3)

Background

In Amazon Simple Storage Service (Amazon S3), uploading a file stores it as an object made of the file data and metadata that describes it.[1] The vast majority of cloud storage on the market uses an object storage architecture, which manages data as objects.[2] Object storage systems allow retention of massive amounts of unstructured data that is written once and read once or many times.[2]

When to use it

In Amazon S3, a bucket is a container for objects, and each object has a key that identifies it uniquely within its bucket.[3] In Amazon S3, uploading an object with a key name that already exists in a versioning-enabled bucket creates another version instead of replacing the existing object.[1] Amazon S3 provides strong read-after-write consistency both for writes of new objects and for requests that overwrite existing objects.[3] Where earlier outputs must survive, put a date or a run identifier in the key, or turn on versioning, rather than writing every run to the same key. (Meridian's view)

In Amazon S3, buckets and the objects in them are private and can be accessed only if access permissions are explicitly granted.[3] Before files can be uploaded to an Amazon S3 bucket, the uploader needs write permissions for the bucket.[1] Grant a pipeline's credentials write access to the one bucket it publishes to, and nothing wider. (Meridian's view)

Object storage is not intended for transactional data, as it does not support the locking and sharing mechanisms needed to keep a single, accurately updated version of a file.[2] Reach for an object store to publish finished outputs that other people and tools read, not as a working file that several processes edit at once. (Meridian's view)

The method

Object storage was proposed at Garth Gibson's Carnegie Mellon University lab as a research project in 1996.[2] Amazon S3 debuted in March 2006.[2] Amazon S3 general purpose buckets have a flat structure instead of a file system's hierarchy, and the Amazon S3 console shows folders by grouping objects whose names share a prefix.[4] Most object storage application programming interfaces (APIs) are based on representational state transfer (REST), so they can use many standard Hypertext Transfer Protocol (HTTP) calls.[2]

DuckDB's httpfs extension reads and writes files on object storage servers through the S3 API, which offers a standard way to write to remote files that regular HTTP servers do not.[5] Services other than Amazon S3 that implement the S3 API, such as Cloudflare R2, should also work with DuckDB's httpfs extension, though not all features may be supported.[5] In DuckDB, writing to S3 works for both comma-separated values (CSV) and Parquet, and uses the multipart upload API.[5]

Amazon S3's multipart upload sends a single object as a set of parts that can be uploaded independently and in any order, and Amazon S3 then assembles the parts into the object.[6] In an Amazon S3 multipart upload, a part whose transmission fails can be retransmitted without affecting the other parts.[6] In Amazon S3, using multipart upload for objects of 100 MB or larger, instead of uploading them in a single operation, is a best practice.[6] Amazon S3 bills for the storage of a multipart upload's parts until the upload is completed or stopped.[6] A pipeline whose uploads can fail midway should stop or clean up its unfinished multipart uploads. (Meridian's view)

Compared with the alternatives

Object storage manages data as objects, whereas file systems manage data as a file hierarchy and block storage manages it as blocks within sectors and tracks.[2] Writing to a local disk is simpler while one machine reads the result; an object store earns its place once the output must be read from elsewhere. (Meridian's view)

Sources

  1. [1]Uploading objects - Amazon Simple Storage Servicedocumentation
  2. [2]Object storage - Wikipediasecondary source
  3. [3]What is Amazon S3? - Amazon Simple Storage Servicedocumentation
  4. [4]Organizing objects in the Amazon S3 console by using folders - Amazon Simple Storage Servicedocumentation
  5. [5]S3 API Support – DuckDBdocumentation
  6. [6]Uploading and copying objects using multipart upload in Amazon S3 - Amazon Simple Storage Servicedocumentation