Publish / database · All operations
Write to a database
Write or bulk load to a database, warehouse or lake.
Status
- In arcform
Not built yet
arcform does not do this yet. Its place on the roadmap
arcform has
ducklake_publishfor DuckLake.- Shape of the result
- the table sent out, with nothing new to look at
- Level
- needs a DuckDB extension
Variants
- Bulk load into a database
Load many rows at once through the database's native loader or COPY, instead of row inserts.
- Delete rows from a table
Remove the rows of a database table that match keys or a condition.
- Replace a table with the result
Create the table from the result, or empty and refill it, so it holds only the new rows.
- Update rows in a table
Change the values of existing rows in a database table, matched by key or condition.
- Publish as a view
Save the query as a view or materialised view in the database instead of writing its rows.
Append rows to a table
Insert the rows into a database or warehouse table, leaving the rows already there.
Export a whole database
Write every table of a database out to a directory of files.
Create or drop the target
Create or remove the catalog, schema, table or collection that the rows are written into.
Write to a non-relational store
Write rows as documents, keys, graph nodes, search entries or vectors to a NoSQL, search, graph or vector database.
What other tools call it
Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).
Alteryx Designer (3)
Calgary Loaderread 2026-09-27Create an indexed Calgary database (.cydb) from any input
Database output connectors (family)read 2026-09-272 tools: Google BigQuery Output, MongoDB Output (deprecated)
Write Data In-DBread 2026-09-27Create or update a table directly in the database from an in-DB stream
Apache DataFusion (15)
CREATE CATALOG / CREATE DATABASEread 2026-09-27Create a catalog
CREATE SCHEMAread 2026-09-27Create a schema under a catalog
CREATE TABLEread 2026-09-27Create an in-memory table from a query or values list
CREATE VIEWread 2026-09-27Create a virtual table from a query
Catalogs: Schemas (CREATE / DROP SCHEMA)read 2026-09-27Create and drop schemas in a catalog
Create and drop tables, including from a query
DELETEread 2026-09-27Remove rows matching a condition from a table; in-memory tables support it
DROP CATALOGread 2026-09-27Remove a registered external catalog
DROP TABLEread 2026-09-27Remove a table from the catalog
DROP VIEWread 2026-09-27Remove a view from the catalog
Data Insert: INSERT INTOread 2026-09-27Insert rows from a query or values into a table
SELECT INTOread 2026-09-27Create an in-memory table from the query result
UPDATEread 2026-09-27Change column values of rows matching a condition; in-memory tables support it
into_viewread 2026-09-27DataFrame method: Turn the DataFrame into a table provider that can be registered as a view
write_tableread 2026-09-27DataFrame method: Execute and write the results into a registered table
Apache Hop (27)
Beam BigQuery Outputread 2026-09-27Write rows to a Google BigQuery table on the Beam engine
Beam Bigtable outputread 2026-09-27Write rows to a Google Cloud Bigtable table on the Beam engine
Bulk load into MSSQLread 2026-09-27Bulk load an existing text file into a SQL Server table
Bulk load into MySQLread 2026-09-27Bulk load a text file into a MySQL table with LOAD DATA
Cassandra Outputread 2026-09-27Write rows to a Cassandra table
CrateDB Bulk Loaderread 2026-09-27Bulk load rows into CrateDB through its HTTP endpoint or COPY
Deleteread 2026-09-27Delete database rows whose keys match each input row
Doris Bulk Loaderread 2026-09-27Bulk load rows into Apache Doris through stream load
LDAP Outputread 2026-09-27Insert, update or delete entries in an LDAP directory
MS SQL Server Bulk Loaderread 2026-09-27Bulk load rows into SQL Server through the JDBC bulk copy API
Microsoft Access Outputread 2026-09-27Write rows to a table in a Microsoft Access database file
MonetDB Bulk Loaderread 2026-09-27Bulk load rows into MonetDB
MongoDB Outputread 2026-09-27Insert, update or upsert documents in a MongoDB collection
MongoDB deleteread 2026-09-27Delete documents from a MongoDB collection
MySQL bulk loaderread 2026-09-27Bulk load rows into MySQL through a FIFO file and LOAD DATA
Neo4j graph outputread 2026-09-27Map fields onto a graph model and write nodes and relationships to Neo4j
Neo4j importread 2026-09-27Run the Neo4j bulk import command over prepared CSV files
Neo4j outputread 2026-09-27Create or merge one or two nodes and a relationship in Neo4j per row
Oracle Bulk Loaderread 2026-09-27Bulk load rows into Oracle through SQL*Loader
PostgreSQL bulk loaderread 2026-09-27Bulk load rows into PostgreSQL through COPY FROM STDIN
Redis Outputread 2026-09-27Write string, hash, set or list values to Redis
Redshift Bulk Loaderread 2026-09-27Bulk load rows into Amazon Redshift through COPY from S3
Snowflake bulk loaderread 2026-09-27Bulk load rows into Snowflake through staged files and COPY
Table Outputread 2026-09-27Insert rows into a database table, with batching and partitioned table names
Teradata Bulk Loaderread 2026-09-27Bulk load rows into Teradata with the fastload utility
Vertica Bulk Loaderread 2026-09-27Bulk load rows into Vertica through a COPY stream
pgvector upsertread 2026-09-27Upsert text chunks and their embeddings into a PostgreSQL pgvector table
Apache NiFi (7)
DatabaseRecordSinkread 2026-09-27Record sink: write records through a database connection
NoSQL, search and analytics store write processors (family)read 2026-09-2717 processors, MongoDB, DynamoDB, Elasticsearch, Cosmos DB, Data Explorer, BigQuery, Redis, Splunk; incl. deletes
PutDatabaseRecordread 2026-09-27Translate incoming records to SQL statements and execute them as a single transaction
PutDistributedMapCacheread 2026-09-27Put FlowFile content into a map cache under a key computed from attributes
PutRecordread 2026-09-27Send records to whichever record sink service is configured
PutSQLread 2026-09-27Execute a SQL update or insert held in FlowFile content, with parameters from attributes
UpdateDatabaseTableread 2026-09-27Generate the table changes a database needs to accept the incoming flat records
Apache SeaTunnel (1)
Sink connectors (family)read 2026-09-2780 connectors, one per system: Jdbc, Kafka, S3File, LocalFile, Doris, Iceberg, Console, Assert, …
AWS Glue (1)
Database connection types as sink (family)read 2026-09-2716 connectionType values that also serve as a sink: redshift, dynamodb, mongodb, postgresql, snowflake, …
Azure Data Factory (5)
Copy activity sink data stores: Azure (family)read 2026-09-2716 data stores usable as copy sink: Azure Blob Storage, Azure Cognitive Search Index, Azure Cosmos DB for NoSQL, …
Copy activity sink data stores: Database (family)read 2026-09-276 data stores usable as copy sink: Informix, Microsoft Access, Oracle, SAP HANA, …
Copy activity sink data stores: Microsoft Fabric (family)read 2026-09-272 data stores usable as copy sink: Microsoft Fabric Lakehouse, Microsoft Fabric Warehouse
Copy activity sink data stores: NoSQL (family)read 2026-09-272 data stores usable as copy sink: MongoDB, MongoDB Atlas
Sinkread 2026-09-27Final destination of a data flow stream
Bruin (8)
DDLread 2026-09-27Create the table once from the column definitions in the asset, with no query
Python assets: materialize()read 2026-09-27Load the dataframe, Arrow table or dict rows a Python function returns into a destination table
appendread 2026-09-27Insert the query result into the table without touching existing rows
create+replaceread 2026-09-27Create the table if absent and overwrite its contents with the query result on every run
ingestr: destinations (family)read 2026-09-2727 destination rows in the strategy table: bigquery, snowflake, postgres, duckdb, clickhouse, …
tableread 2026-09-27Materialize the asset query result as a table
truncate+insertread 2026-09-27Truncate the table, keeping its structure, and insert the full query result
viewread 2026-09-27Materialize the asset query as a view
dbt-utils (4)
appendread 2026-09-27Insert selected rows into the target without updating, deleting or checking duplicates
materialized_viewread 2026-09-27Create and maintain a materialized view that the platform refreshes
tableread 2026-09-27Rebuild the model as a table on each run via create table as
viewread 2026-09-27Rebuild the model as a view on each run via create view as
DuckDB (1)
EXPORT and IMPORT DATABASEread 2026-09-27Write a whole database to a directory of files, and read it back
Duckle (16)
Azure Synapseread 2026-09-25write to Azure Synapse through DuckDB mssql extension
BigQueryread 2026-09-25write to BigQuery through DuckDB bigquery extension
Cloud Warehouses sinks outside DuckDB (family)read 2026-09-253 components, one per system, run by Rust drivers and HTTP clients, outside DuckDB: gizmosql, snowflake, databricks
Databases sinks outside DuckDB (family)read 2026-09-256 components, one per system, run by Rust drivers and HTTP clients, outside DuckDB: oracle, teradata, db2, access, turso, …
Databases sinks via the mysql extension (family)read 2026-09-252 components, one per system, run by DuckDB mysql extension: mysql, mariadb
Databases sinks via the postgres extension (family)read 2026-09-252 components, one per system, run by DuckDB postgres extension: postgres, cockroach
DuckDBread 2026-09-25write a table into an attached DuckDB file
DuckDB Quackread 2026-09-25write to DuckDB Quack through DuckDB quack extension
DuckLakeread 2026-09-25write a table into an attached DuckLake catalog
MotherDuckread 2026-09-25write to MotherDuck through DuckDB motherduck extension
NoSQL & Search sinks outside DuckDB (family)read 2026-09-258 components, one per system, run by Rust drivers and HTTP clients, outside DuckDB: mongodb, cassandra, scylla, redis, neo4j, …
Redshiftread 2026-09-25write to Redshift through DuckDB postgres extension
SQL Serverread 2026-09-25write to SQL Server through DuckDB mssql extension
SQLiteread 2026-09-25write a table into an attached SQLite file
Vector / AI Databases sinks outside DuckDB (family)read 2026-09-256 components, one per system, run by Rust drivers and HTTP clients, outside DuckDB: pinecone, qdrant, weaviate, milvus, …
pgvector (Postgres)read 2026-09-25write to pgvector (Postgres) through DuckDB postgres extension
EasyMorph (5)
Bulk export to databaseread 2026-09-27Bulk load the table into a database table, on databases that support it (paid editions only)
Delete database rowsread 2026-09-27Delete the rows of a database table that satisfy query conditions
Delete matching database rowsread 2026-09-27Delete database rows whose keys match, or do not match, keys in an EasyMorph table
Export to databaseread 2026-09-27Insert the table's rows into a database table
Update database tableread 2026-09-27Update rows of a database table with values from the EasyMorph table (paid editions only)
KNIME Analytics Platform (9)
DB Loaderread 2026-09-27Bulk load a table into an existing database table using the database's native loader
DB Reference Row Deleterread 2026-09-27Delete database rows whose key columns match rows of the input table
DB Row Deleterread 2026-09-27Delete rows of a database table that match configured filter criteria
DB Row Inserterread 2026-09-27Insert input-table rows into a database table
DB Table Deleterread 2026-09-27Drop a table from the database
DB Table Structure Creatorread 2026-09-27Create a new database table from a manual definition or an input table's spec
DB Updaterread 2026-09-27Update database rows identified by key columns with values from the input table
DB Writerread 2026-09-27Write a table to a database table, creating the table when it does not exist
DB Writer (DB Data)read 2026-09-27Materialise the result of a DB query as a new table inside the database
Pentaho Data Integration (26)
Cassandra Outputread 2026-09-27Write rows to a Cassandra column family
Deleteread 2026-09-27Delete rows from a database table
ElasticSearch Bulk Insert (deprecated)read 2026-09-27Deprecated Elasticsearch bulk writer, replaced by Elasticsearch REST Bulk Insert
Elasticsearch REST Bulk Insertread 2026-09-27Bulk insert rows into Elasticsearch
Greenplum Bulk Loader (deprecated)read 2026-09-27Deprecated Greenplum bulk loader, replaced by Greenplum Load
Greenplum Loadread 2026-09-27Bulk load rows into Greenplum
HBase Outputread 2026-09-27Write rows to an HBase column family
Infobright Loaderread 2026-09-27Bulk load rows into an Infobright table
Ingres VectorWise Bulk Loaderread 2026-09-27Bulk load rows into Ingres VectorWise through its COPY TABLE command
LDAP Outputread 2026-09-27Insert, upsert, update, add or delete LDAP records by distinguished name
LucidDB streaming loader (deprecated)read 2026-09-27Deprecated: load rows into LucidDB through its Remote Rows UDX
Microsoft Access Outputread 2026-09-27Write rows to a table of a Microsoft Access database
MonetDB Agile Martread 2026-09-27Load rows into a MonetDB agile mart; the reference gives the step no description
MonetDB Bulk Loaderread 2026-09-27Bulk load rows into MonetDB in streaming mode
MongoDB Outputread 2026-09-27Write rows to a MongoDB collection
MySQL Bulk Loaderread 2026-09-27Bulk load rows into MySQL over a named pipe
Oracle Bulk Loaderread 2026-09-27Bulk load rows into Oracle with Oracle Bulk Loader
Palo cell output (deprecated)read 2026-09-27Deprecated: update cell data in a Palo OLAP cube
Palo dim output (deprecated)read 2026-09-27Deprecated: create or update dimension elements and consolidations in Palo
PostgreSQL Bulk Loaderread 2026-09-27Bulk load rows into PostgreSQL
Splunk Outputread 2026-09-27Write data to Splunk
Table Agile Martread 2026-09-27Load rows into an agile mart table; the reference gives the step no description
Table Outputread 2026-09-27Write rows to a database table
Teradata Fastload Bulk Loaderread 2026-09-27Bulk load rows into Teradata with Fastload
Teradata TPT Insert Upsert Bulk Loaderread 2026-09-27Bulk load or upsert rows into Teradata through TPT and the tbuild command
Vertica Bulk Loaderread 2026-09-27Bulk load rows into a Vertica table through COPY
Polars (1)
DataFrame.write_databaseread 2026-09-27Write a frame to a database table
RapidMiner Studio (16)
Add to Solr (Data)read 2026-09-27Index the rows of a table into a Solr collection
Add to Solr (Documents)read 2026-09-27Index a collection of text documents into Solr
Create Collection (Milvus)read 2026-09-27Create a collection in a Milvus vector database
Create Collection (Qdrant)read 2026-09-27Create a collection in a Qdrant vector database
Delete Cassandraread 2026-09-27Delete rows from a Cassandra or Astra table by the input ID column
Delete Collection (Milvus)read 2026-09-27Delete a collection from a Milvus vector database
Delete Collection (Qdrant)read 2026-09-27Delete a collection from a Qdrant vector database
Delete MongoDBread 2026-09-27Delete documents from a MongoDB collection
Insert Documents (Milvus)read 2026-09-27Insert rows with embeddings as documents into a Milvus collection
Insert Documents (Qdrant)read 2026-09-27Insert rows with embeddings as documents into a Qdrant collection
Store (In Database)read 2026-09-27Store the result of an in-database chain as a database table
Write Accessread 2026-09-27Write a table into a Microsoft Access database file
Write Cassandraread 2026-09-27Write a table to a Cassandra or Astra table
Write Databaseread 2026-09-27Write a table to a SQL database table
Write Google BigQueryread 2026-09-27Write a table to a Google BigQuery table
Write MongoDBread 2026-09-27Write documents to a MongoDB collection
SQLMesh (5)
FULLread 2026-09-27Rewrite the whole table on every model evaluation
INCREMENTAL_UNMANAGEDread 2026-09-27Run the query on schedule and append whatever it returns to the table
MANAGEDread 2026-09-27Create a table whose data the database engine itself keeps up to date
VIEWread 2026-09-27Create or replace a non-materialized view; the default kind
VIEW (materialized)read 2026-09-27VIEW with materialized true, on engines that support materialized views
Tableau Prep (1)
Database tableread 2026-09-27Write to an external database table: append, create or replace, with SQL before and after
Talend (10)
Database bulk load components (family)read 2026-09-2768 components writing bulk files or running bulk loaders: tMysqlOutputBulk, tOracleBulkExec, tTeradataFastLoad
Database output components (family)read 2026-09-2763 components, one per database, inserting, updating or deleting rows: tMysqlOutput, tOracleOutput, tSnowflakeOutput
tHBaseDeleteRowsread 2026-09-27Delete rows from an HBase table by row key
tLDAPRenameEntryread 2026-09-27Rename one or more entries in an LDAP directory
tMatchIndexread 2026-09-27Index a clean, deduplicated data set in Elasticsearch for continuous matching
tNeo4jImportToolread 2026-09-27Create a Neo4j database and bulk import CSV files into it with the Neo4j Import Tool
tNeo4jv4ImportToolread 2026-09-27Create a Neo4j 4.x database and bulk import CSV files into it with the Neo4j Import Tool
tPineconeClientread 2026-09-27Upsert, update or delete records in a Pinecone index namespace, or query and fetch them
tSqoopExport (deprecated)read 2026-09-27Call Sqoop to transfer data from HDFS to a relational database
tSynonymOutputread 2026-09-27Create a Lucene index and feed it with entries and their synonyms
Background
A data manipulation language (DML) is used for adding (inserting), deleting and modifying (updating) data in a database.[1] In Structured Query Language (SQL), DML statements are organised by their first word, a verb such as INSERT INTO, UPDATE or DELETE FROM.[1] In extract, transform, load (ETL), the load phase puts the data into its end target, which can be any data store, from a flat file to a data warehouse.[2]
When to use it
Some data warehouses may overwrite existing information with cumulative information, while others may add new data in a historical form at regular intervals.[2] More complex ETL systems can keep a history and audit trail of all changes to the data loaded into the data warehouse.[2] In ETL, when to replace and when to append are strategic design choices that depend on the time available and the business needs.[2] In Bruin, the append strategy does not deduplicate, so running the same time window twice inserts the rows twice.[3] Decide before the first run whether a run adds to its target or replaces it, and what a second run over the same data should leave behind. (Meridian's view)
When the load phase writes to a database, the constraints in the database schema, such as uniqueness, referential integrity and mandatory fields, apply, as do triggers activated on load.[2] In SQLite, the datatype of a value is associated with the value itself, not with the column it is stored in.[4] SQLite has provided STRICT tables, which enforce types rigidly, since version 3.37.0 (2021-11-27).[4] Check what a target database enforces before relying on it to reject a bad row. (Meridian's view)
In PostgreSQL, a row can be reliably addressed only through a primary key, because SQL does not in general provide a unique identifier for rows.[5] Give a table a key before writing to it in any way other than appending, so that a change can name the rows it means. (Meridian's view)
The method
In SQL, statements that create or change tables and other database objects, rather than the data in them, belong to a separate data definition language (DDL).[1] In DuckDB, INSERT INTO adds new rows to a table, either rows given as value expressions or the rows a query returns.[6] In DuckDB, INSERT INTO ... BY NAME matches the column names of a SELECT query against the table's, so the columns may come in a different order or some may be missing.[6] In DuckDB, a column left out of an insert is filled with its declared default value, or NULL if it has none.[6]
In DuckDB, the ATTACH statement adds a database file to the catalogue, where it can be read from and written to.[7] In DuckDB, ATTACH definitions are not kept between sessions, so each new session has to attach its databases again.[7] DuckDB's SQLite extension can create SQLite database files, create tables and ingest data into SQLite with standard SQL queries.[8]
Compared with the alternatives
DuckDB's EXPORT DATABASE command exports the contents of a database to a directory, and its IMPORT DATABASE command reads them back.[9]
Sources
- [1]Data manipulation language - Wikipediasecondary source
- [2]Extract, transform, load - Wikipediasecondary source
- [3]Materialization | Bruin CLIdocumentation
- [4]Datatypes In SQLitedocumentation
- [5]PostgreSQL: Documentation: 18: 6.2. Updating Datadocumentation
- [6]INSERT Statement – DuckDBdocumentation
- [7]ATTACH and DETACH Statements – DuckDBdocumentation
- [8]SQLite Extension – DuckDBdocumentation
- [9]EXPORT and IMPORT DATABASE Statements – DuckDBdocumentation