Publish / database · All operations

Bulk load into a database

A variant of Write to a database

Load many rows at once through the database's native loader or COPY, instead of row inserts.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache Hop (14)
EasyMorph (1)
  • Bulk export to databaseread 2026-09-27

    Bulk load the table into a database table, on databases that support it (paid editions only)

KNIME Analytics Platform (1)
  • DB Loaderread 2026-09-27

    Bulk load a table into an existing database table using the database's native loader

Pentaho Data Integration (12)
Talend (4)
  • Database bulk load components (family)read 2026-09-27

    68 components writing bulk files or running bulk loaders: tMysqlOutputBulk, tOracleBulkExec, tTeradataFastLoad

  • tNeo4jImportToolread 2026-09-27

    Create a Neo4j database and bulk import CSV files into it with the Neo4j Import Tool

  • tNeo4jv4ImportToolread 2026-09-27

    Create a Neo4j 4.x database and bulk import CSV files into it with the Neo4j Import Tool

  • tSqoopExport (deprecated)read 2026-09-27

    Call Sqoop to transfer data from HDFS to a relational database

Background

In PostgreSQL, the COPY command loads all the rows in one command and is optimised for loading large numbers of rows.[1] In PostgreSQL, loading many rows with COPY is almost always faster than with INSERT statements, even with prepared statements batched into one transaction.[1] In DuckDB, the Appender loads bulk data into a database from the C, C++, Go, Java, Julia, Node.js, Python and Rust clients.[2]

When to use it

In DuckDB, row-by-row inserts, even with prepared statements, are detrimental to performance and result in slow load times.[3] In DuckDB, inserting in loops is a method to avoid unless the data is small (under 100k rows).[3] In EasyMorph, the Bulk export to database action is meant for large volumes, such as millions of rows, and may not be efficient on a few hundred rows.[4] Reach for a bulk loader when the load is large, and keep plain inserts for a handful of rows. (Meridian's view)

In KNIME, the DB Loader node uses bulk loading functionality that only some databases, such as Hive, MySQL, PostgreSQL and H2, support.[5] In KNIME's DB Loader, most databases perform no data checks as the data is loaded, which might lead to a corrupt data table.[5] KNIME's DB Loader checks that column order and column names are compatible, but not that column types are.[5] With Apache Hop's PostgreSQL bulk loader, COPY error messages rarely name the field and value at fault.[6] Validate the rows before they reach a bulk loader. (Meridian's view)

In DuckDB, an Appender checks constraints such as NOT NULL, PRIMARY KEY and UNIQUE when its cache is flushed, so a row can be accepted and later fail at flush time.[2]

The method

In PostgreSQL, COPY FROM copies data from a file to a table, appending it to whatever is in the table already.[7] In DuckDB, COPY ... FROM imports data from an external file into an existing table, appending it to the data already there.[8] Apache Hop's PostgreSQL bulk loader streams data into PostgreSQL with COPY DATA FROM STDIN.[6]

In DuckDB, an Appender is tied to a single connection, always appends to a single table, and caches values before inserting them into the database.[2] In KNIME's DB Loader, an intermediate file format is often used for efficiency, depending on the database.[5]

In PostgreSQL, the fastest way to fill a freshly created table is to create it, bulk load it with COPY, and then create the indexes it needs.[1] In PostgreSQL, a foreign key constraint can be checked in bulk more efficiently than row by row, so it might be useful to drop it, load the data and re-create it, trading error checking for load speed while it is missing.[1]

Compared with the alternatives

In PostgreSQL, COPY is less flexible than INSERT but incurs significantly less overhead for large data loads.[1] In PostgreSQL, where COPY cannot be used, a prepared INSERT executed many times avoids some of the overhead of parsing and planning each INSERT again.[1] In PostgreSQL, committing many INSERTs in one transaction saves work per row, and a failure then rolls back every row inserted so far rather than leaving a partial load.[1] In DuckDB's auto-commit mode, every INSERT runs in a separate transaction and is synced to disk, which significantly slows a load made of many inserts.[9]

Sources

  1. [1]PostgreSQL: Documentation: 18: 14.4. Populating a Databasedocumentation
  2. [2]Appender – DuckDBdocumentation
  3. [3]Data Import – DuckDBdocumentation
  4. [4]BULK EXPORT TO DATABASE [EasyMorph Help]documentation
  5. [5]DB Loaderdocumentation
  6. [6]PostgreSQL bulk loader | Apache Hopdocumentation
  7. [7]PostgreSQL: Documentation: 18: COPYdocumentation
  8. [8]COPY Statement – DuckDBdocumentation
  9. [9]INSERT Statements – DuckDBdocumentation