Join / diff · All operations
Report the differences between two tables
A variant of Compare two tables
List where two tables differ, cell by cell, column by column or as summary statistics.
Status
- In arcform
Runs as a SQL step
arcform carries it as a
sql:step today.
The DuckDB step
Report differences cell by cell, column by column and as summary figures
CREATE TEMP TABLE v1 AS SELECT * FROM (VALUES
(1, 'ann', 10, 1.5), (2, 'bob', 5, NULL), (3, 'cy', 7, 2.0), (4, 'di', 1, 3.0), (6, 'fay', 3, 1.0), (NULL, 'nk', 1, 1.0)) t(id, name, qty, price);
CREATE TEMP TABLE v2 AS SELECT * FROM (VALUES
(1, 'ann', 12, 1.5), (2, 'bob', 5, 2.5), (3, 'cyd', 8, 2.0), (5, 'ed', 2, 4.0), (6, 'fay', 3, 1.0), (NULL, 'nk', 1, 1.0)) t(id, name, qty, price);
-- the key must be unique on each side, or a record is compared more than once; a NULL key matches nothing, not even itself
CREATE TEMP TABLE repeated_keys AS
SELECT 'v1' AS side, id FROM v1 WHERE id IS NOT NULL GROUP BY id HAVING count(*) > 1
UNION ALL SELECT 'v2', id FROM v2 WHERE id IS NOT NULL GROUP BY id HAVING count(*) > 1;
-- every column as text, one row per key and column
CREATE TEMP TABLE old_cells AS
FROM (SELECT id, name::VARCHAR AS name, qty::VARCHAR AS qty, price::VARCHAR AS price FROM v1)
UNPIVOT INCLUDE NULLS (val FOR col IN (name, qty, price));
CREATE TEMP TABLE new_cells AS
FROM (SELECT id, name::VARCHAR AS name, qty::VARCHAR AS qty, price::VARCHAR AS price FROM v2)
UNPIVOT INCLUDE NULLS (val FOR col IN (name, qty, price));
-- cell by cell: one row for each cell that differs; IS DISTINCT FROM also sees NULL turning into a value
CREATE TEMP TABLE cell_report AS
SELECT id, col, o.val AS old_value, n.val AS new_value
FROM old_cells o JOIN new_cells n USING (id, col) WHERE o.val IS DISTINCT FROM n.val;
-- column by column: one row for each column compared
CREATE TEMP TABLE column_report AS
SELECT col, count(*) AS compared, count(*) FILTER (WHERE o.val IS DISTINCT FROM n.val) AS differing
FROM old_cells o JOIN new_cells n USING (id, col) GROUP BY col;
-- summary figures: one row for the two tables
CREATE TEMP TABLE summary AS
SELECT (SELECT count(*) FROM v1) AS rows_v1, (SELECT count(*) FROM v2) AS rows_v2,
count(*) AS matched, count(*) FILTER (WHERE a IS DISTINCT FROM b) AS changed, count(*) FILTER (WHERE a IS NOT DISTINCT FROM b) AS identical,
(SELECT count(*) FROM v1 ANTI JOIN v2 ON v1.id = v2.id) AS only_in_v1, (SELECT count(*) FROM v2 ANTI JOIN v1 ON v1.id = v2.id) AS only_in_v2
FROM v1 a JOIN v2 b ON a.id = b.id;
SELECT * FROM cell_report ORDER BY id, col;
SELECT * FROM column_report ORDER BY col;
SELECT * FROM summary;
SELECT CASE WHEN ((SELECT count(*) = 0 FROM repeated_keys)
AND (SELECT list((id, col, old_value, new_value) ORDER BY id, col) IS NOT DISTINCT FROM [(1, 'qty', '10', '12'), (2, 'price', NULL, '2.5'), (3, 'name', 'cy', 'cyd'), (3, 'qty', '7', '8')] FROM cell_report)
AND (SELECT map_from_entries(list((col, differing) ORDER BY col)) = MAP {'name': 1, 'price': 1, 'qty': 2} AND min(compared) = 4 AND max(compared) = 4 FROM column_report)
AND (SELECT rows_v1 = 6 AND rows_v2 = 6 AND matched = 4 AND changed = 3 AND identical = 1 AND only_in_v1 = 2 AND only_in_v2 = 2 FROM summary)) THEN 'as expected' ELSE error('the report should hold 4 cells (qty of 1 and 3, the price of 2 turning from NULL to 2.5, the name of 3), 4 compared records in each column, and 4 matched records of which 3 changed, with ids 4 and 5 and the two NULL-key rows each unmatched') END AS verdict;- The statement builds its own data.
- It needs nothing installed beyond DuckDB itself.
- The statement checks its own result and raises an error when it is wrong: “the report should hold 4 cells (qty of 1 and 3, the price of 2 turning from NULL to 2.5, the name of 3), 4 compared records in each column, and 4 matched records of which 3 changed, with ids 4 and 5 and the two NULL-key rows each unmatched”.
- It passed on DuckDB
v2.0.0 (Cyanoptera) d54cbbe3df.
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
Apache Hop (1)
Table Compareread 2026-09-27Compare the contents of two database tables and report the differences
Bruin (1)
data-diffread 2026-09-27Compare two tables across connections: schema by default, per-column statistics with --full
Duckle (1)
Reconcile (Source vs Target)read 2026-09-25source against target reconciliation report joined on keys
EasyMorph (1)
Compare tablesread 2026-09-27Compare two tables and list the differences found per column
KNIME Analytics Platform (1)
Table Difference Finderread 2026-09-27Compare two tables cell by cell and by column spec, reporting the differences
Pentaho Data Integration (1)
Table Compareread 2026-09-27Compare the data of two same-layout tables and log the differences
VisiData (1)
setdiff-sheetread 2026-09-23(experimental) Set this sheet as the diff baseline; a colorizer marks cells that differ from it
Background
In pandas, DataFrame.compare compares two DataFrames element-wise and returns a DataFrame highlighting the differences.[1] KNIME's Table Difference Finder shows value differences for each row and column, and compares column types, domains and positions for each column.[2]
When to use it
EasyMorph's Compare tables action is for comparing the current version of a dataset against an earlier one to determine if it has changed, and what those changes were.[3] In EasyMorph, Compare tables returns an empty dataset when both tables are exactly the same.[3] Bruin's data-diff command is meant for comparing tables between development, staging and production, and for validating data migrations.[4]
In pandas, DataFrame.compare can only compare DataFrames with the same shape and identical row and column labels.[1] Apache Hop's Table Compare needs the two tables to have the same layout.[5] In Bruin, data-diff treats values whose percentage difference is below a tolerance, 0.001 by default, as equal.[4] Set a tolerance on purpose for floating-point columns, and report it with the result. (Meridian's view)
The method
SQLMesh's table diff runs a schema diff, which finds fields added, removed or changed in data type, and a row diff of data values.[6] SQLMesh's table diff reports how many rows are in both tables and in only one, and the percentage of matching values in each compared column of the joined rows.[6]
By default, Bruin's data-diff compares only schemas, and its --full flag adds row counts and per-column statistics such as null count, minimum and maximum.[4] EasyMorph's Compare tables reports differences per column, such as columns added or removed, values that have become or stopped being unique, and a changed number of rows.[3]
Apache Hop's Table Compare outputs a summary of the differences found, and each single difference through its error handling output.[5] In pandas, DataFrame.compare shows equal values as NaN unless keep_equal is set, and matching NaNs do not appear as a difference.[1]
Sources
- [1]pandas.DataFrame.compare — pandas 3.1.1 documentationdocumentation
- [2]Table Difference Finderdocumentation
- [3]COMPARE TABLES [EasyMorph Help]documentation
- [4]data-diff Command | Bruin CLIdocumentation
- [5]Table Compare | Apache Hopdocumentation
- [6]sqlmesh/docs/guides/tablediff.md at 23b237cea7218cc8f338b89c14f0957394a0003b · SQLMesh/sqlmesh · GitHubdocumentation