Join / diff · All operations
Compare two tables
Compare two tables, report changed, added and removed rows.
Status
- In arcform
Runs as a SQL step
arcform carries it as a
sql:step today.- Shape of the result
- one table from two or more
- Level
- several clauses of SQL, or a window
The DuckDB step
Compare two versions of a table by key
It reads the orders table the proofs share. Create it first:
CREATE TABLE orders AS SELECT * FROM (VALUES
(1, 'Alice Ng', 'north', DATE '2026-01-03', 120.00, 'widget', 'a|b', '{"channel":"web","items":[1,2]}'),
(2, 'alice ng', 'north', DATE '2026-01-05', 80.50, 'gadget', 'b', '{"channel":"shop","items":[3]}'),
(3, 'Bob Okafor', 'south', DATE '2026-01-05', 300.00, 'widget', 'a|c', '{"channel":"web","items":[4,5,6]}'),
(4, 'Carla Díaz', 'east', DATE '2026-01-09', NULL, 'gizmo', NULL, '{"channel":"web","items":[]}'),
(5, 'Bob Okafor', 'south', DATE '2026-01-12', 45.25, 'gadget', 'c', '{"channel":"phone","items":[7]}'),
(6, 'Dev Patel', 'west', DATE '2026-01-12', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'),
(7, 'Carla Diaz', 'east', DATE '2026-01-20', 99.99, 'gizmo', 'b|c', '{"channel":"shop","items":[10]}'),
(8, 'Dev Patel', 'west', DATE '2026-02-02', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'),
(9, 'Eve Larsen', NULL, DATE '2026-02-03', 15.00, 'gadget', NULL, '{"channel":"web","items":[11]}'),
(10, 'Alice Ng', 'north', DATE '2026-02-10', 220.00, 'gizmo', 'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);Then run the statement:
CREATE TEMP TABLE result AS
WITH v2 AS (SELECT * REPLACE (CASE WHEN order_id = 3 THEN 310.0 ELSE amount END AS amount) FROM orders WHERE order_id <> 9
UNION ALL SELECT 11, 'Gus Hale', 'north', DATE '2026-02-11', 5.0, 'gadget', NULL, '{}')
SELECT coalesce(a.order_id, b.order_id) AS order_id,
CASE WHEN a.order_id IS NULL THEN 'added' WHEN b.order_id IS NULL THEN 'removed'
WHEN a IS DISTINCT FROM b THEN 'changed' ELSE 'same' END AS status
FROM orders a FULL JOIN v2 b USING (order_id)
WHERE status <> 'same' ORDER BY 1;
SELECT * FROM result;
SELECT CASE WHEN ((SELECT map_from_entries(list((order_id, status) ORDER BY order_id)) = MAP {3: 'changed', 9: 'removed', 11: 'added'} FROM result)) THEN 'as expected' ELSE error('the difference is not order 3 changed, 9 removed and 11 added') END AS verdict;- It needs nothing installed beyond DuckDB itself.
- The statement checks its own result and raises an error when it is wrong: “the difference is not order 3 changed, 9 removed and 11 added”.
- It passed on DuckDB
v2.0.0 (Cyanoptera) d54cbbe3df.
Variants
- Check that two tables are identical
Return whether two tables hold the same data.
- Report the differences between two tables
List where two tables differ, cell by cell, column by column or as summary statistics.
Flag added, changed and removed rows
Match the rows of two tables on a key and tag each as unchanged, changed, added or removed.
What other tools call it
Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).
Apache Hop (2)
Merge rows (diff)read 2026-09-27Compare a reference and a new sorted stream, flagging rows identical, changed, new or deleted
Table Compareread 2026-09-27Compare the contents of two database tables and report the differences
AWS Glue (1)
DatasetMatchread 2026-09-27Compare a dataset with a reference dataset and tell whether they are in sync
Bruin (1)
data-diffread 2026-09-27Compare two tables across connections: schema by default, per-column statistics with --full
Duckle (3)
Diff Detectread 2026-09-25tag inserted, updated and deleted rows against a previous snapshot
DuckLake Data Diffread 2026-09-25row-level change feed between two chosen snapshots of a DuckLake table
Reconcile (Source vs Target)read 2026-09-25source against target reconciliation report joined on keys
EasyMorph (1)
Compare tablesread 2026-09-27Compare two tables and list the differences found per column
KNIME Analytics Platform (1)
Table Difference Finderread 2026-09-27Compare two tables cell by cell and by column spec, reporting the differences
Pentaho Data Integration (2)
Merge Rows (diff)read 2026-09-27Compare two sorted streams and flag rows equal, changed, deleted or new
Table Compareread 2026-09-27Compare the data of two same-layout tables and log the differences
Polars (1)
DataFrame.equalsread 2026-09-27Test whether two frames hold the same data
SQLMesh (1)
table_diffread 2026-09-27Schema diff and outer-join row diff of two tables, or one model across two environments
VisiData (1)
setdiff-sheetread 2026-09-23(experimental) Set this sheet as the diff baseline; a colorizer marks cells that differ from it
Background
In Apache Hop, Merge Rows (diff) matches previous and new rows on key fields and flags each as identical, changed, new or deleted.[1] In Apache Hop's Merge Rows (diff), a changed row is one whose key is found in both inputs but one or more of its compared values differ.[1] A full outer join combines the effect of left and right outer joins, giving NULL in every column of the table that lacks a matching row.[2]
When to use it
Apache Hop's Merge Rows (diff) is useful for comparing data collected at two different times.[1] In Apache Hop, when a source has no change log and no reliable timestamp or increasing ID, change data capture (CDC) is a comparison of two full extracts.[3] Reach for a full comparison when the question is what happened between two versions of a table, and a timestamp column cannot be trusted to say. (Meridian's view)
Apache Hop's Table Compare needs the key fields of the tables, because without them the two tables cannot be correctly joined.[4] In SQLMesh, the grain that joins the two tables in a table diff should be unique and not null, and a warning is shown when rows contain null or duplicate grains.[5]
In SQL, a comparison with NULL can never result in true or false, but always in a third logical result, Unknown.[6] In DuckDB, the = operator returns NULL when either input is NULL, while IS DISTINCT FROM treats NULL values as equal.[7]
The method
Apache Hop's Merge Rows (diff) is a streaming compare that does not sort, so both inputs must already be ordered on the keys to match.[3] In Apache Hop, if either stream into Merge Rows (diff) is out of order, the flags are wrong.[3] Apache Hop's Merge Rows (diff) walks both sorted streams once and keeps only the current pair of rows in memory.[3]
Compared with the alternatives
The SQL EXCEPT operator returns the distinct rows of one query that do not appear in a second result set.[8] In databases, change data capture (CDC) is a set of software design patterns used to determine and track the data that has changed.[9]
Sources
- [1]Merge rows (diff) | Apache Hopdocumentation
- [2]Join (SQL) - Wikipediasecondary source
- [3]CDC: comparing snapshots | Apache Hopdocumentation
- [4]Table Compare | Apache Hopdocumentation
- [5]sqlmesh/docs/guides/tablediff.md at 23b237cea7218cc8f338b89c14f0957394a0003b · SQLMesh/sqlmesh · GitHubdocumentation
- [6]Null (SQL) - Wikipediasecondary source
- [7]Comparisons – DuckDBdocumentation
- [8]Set operations (SQL) - Wikipediasecondary source
- [9]Change data capture - Wikipediasecondary source