Join / merge upsert · All operations
Insert only new keys
A variant of Merge into a table
Insert a source row only when its key, or its content hash, is not already in the target, leaving existing rows untouched.
Status
- In arcform
Runs as a SQL step
arcform carries it as a
sql:step today.
The DuckDB step
Insert a source row only when its key, or its content hash, is new
CREATE TEMP TABLE source AS SELECT * FROM (VALUES
(1, 'a', 1), (1, 'a2', 1), (3, 'c', 3), (3, 'c', 3), (4, 'b', 2), (NULL, 'n', 0), (5, NULL, 5)) t(id, name, qty);
-- by key: existing keys are left alone (key 1 keeps its values); the source keeps one row for each key, and a NULL key is not a key
CREATE TEMP TABLE by_key AS SELECT * FROM (VALUES (1, 'a', 1), (2, 'b', 2)) t(id, name, qty);
MERGE INTO by_key t USING (SELECT id, name, qty FROM source WHERE id IS NOT NULL QUALIFY row_number() OVER (PARTITION BY id ORDER BY name) = 1) s ON t.id = s.id
WHEN NOT MATCHED THEN INSERT VALUES (s.id, s.name, s.qty);
-- by content hash: a row is new when no target row has the same hash of its values; the key plays no part. The hash is SHA-256 of the row as JSON,
-- which does not change between DuckDB versions, so it can be stored; a NULL is 'null' in the JSON and an empty string is '""', so they differ
CREATE TEMP TABLE by_hash AS SELECT name, qty, sha256(to_json({'name': name, 'qty': qty})) AS row_hash FROM (VALUES ('a', 1), ('b', 2)) t(name, qty);
MERGE INTO by_hash t
USING (SELECT name, qty, sha256(to_json({'name': name, 'qty': qty})) AS row_hash FROM source QUALIFY row_number() OVER (PARTITION BY row_hash ORDER BY id) = 1) s ON t.row_hash = s.row_hash
WHEN NOT MATCHED THEN INSERT VALUES (s.name, s.qty, s.row_hash);
SELECT * FROM by_key ORDER BY id;
SELECT name, qty FROM by_hash ORDER BY name NULLS LAST, qty;
SELECT CASE WHEN ((SELECT list((id, name, qty) ORDER BY id) = [(1, 'a', 1), (2, 'b', 2), (3, 'c', 3), (4, 'b', 2), (5, NULL, 5)] FROM by_key)
AND (SELECT list((name, qty) ORDER BY name NULLS LAST, qty) IS NOT DISTINCT FROM [('a', 1), ('a2', 1), ('b', 2), ('c', 3), ('n', 0), (NULL, 5)] FROM by_hash)) THEN 'as expected' ELSE error('by key the target should gain keys 3 (once), 4 and 5 and keep key 1 as it was; by content hash it should gain a2, c, n and the NULL-name row, and not the repeat of a, or the row with key 4 whose values b and 2 are already there') END AS verdict;- The statement builds its own data.
- It needs nothing installed beyond DuckDB itself.
- The statement checks its own result and raises an error when it is wrong: “by key the target should gain keys 3 (once), 4 and 5 and keep key 1 as it was; by content hash it should gain a2, c, n and the NULL-name row, and not the repeat of a, or the row with key 4 whose values b and 2 are already there”.
- It passed on DuckDB
v2.0.0 (Cyanoptera) d54cbbe3df.
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
Apache Hop (1)
Combination lookup/updateread 2026-09-27Look up or insert a junk-dimension row for a combination of attributes, returning its key
Bruin (3)
datavault_hubread 2026-09-27Insert one row per hub hash key, earliest load first, only when the key is new
datavault_linkread 2026-09-27Insert one row per link hash key, earliest load first, only when the key is new
datavault_satelliteread 2026-09-27Insert a row per parent hash key only when its hashdiff differs from the latest stored row
Pentaho Data Integration (1)
Combination lookup/updateread 2026-09-27Look up or insert a junk-dimension row keyed on all of its fields
Background
In MySQL, INSERT IGNORE ignores duplicate key errors and goes on, so existing rows are neither inserted nor updated while all new rows are inserted.[1] In DuckDB, INSERT OR IGNORE is a shorter form of INSERT INTO ... ON CONFLICT DO NOTHING, under which conflicting values are not inserted or updated.[2]
When to use it
In a data vault, hubs contain a list of unique business keys with low propensity to change, each with a surrogate key and metadata on the key's origin.[3] In Apache Hop, Combination lookup/update looks up a combination of business key fields, returns its technical key if it exists, and otherwise inserts a row with a new technical key.[4] Choose this variant only when a later row for a known key carries nothing you need to keep. (Meridian's view)
In DuckDB, an ON CONFLICT clause acts on conflicts that arise from UNIQUE or PRIMARY KEY constraints.[2] Where the target carries no such constraint, test for the key with a MERGE or a NOT EXISTS instead. (Meridian's view) Decide what a missing key means before loading, since a row with no key cannot be told apart from the rows already there. (Meridian's view)
The method
Data vault modelling, a database modelling method published in 2000 by Dan Linstedt, is designed to provide long-term historical storage of data from multiple operational systems.[3] In a data vault, every row must be accompanied by record source and load date attributes, so an auditor can trace values back to the source.[3]
Compared with the alternatives
MySQL's REPLACE INTO first attempts an insert and, if that fails, deletes the existing row and inserts the new one.[1] In DuckDB, INSERT OR REPLACE is a shorter form of INSERT INTO ... DO UPDATE SET, setting each column to the value of the row being inserted.[2]
Sources
- [1]Merge (SQL) - Wikipediasecondary source
- [2]INSERT Statement – DuckDBdocumentation
- [3]Data vault modeling - Wikipediasecondary source
- [4]Combination lookup/update | Apache Hopdocumentation