Join / diff · All operations

Report the differences between two tables

A variant of Compare two tables

List where two tables differ, cell by cell, column by column or as summary statistics.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

The DuckDB step

Report differences cell by cell, column by column and as summary figures

SQL
CREATE TEMP TABLE v1 AS SELECT * FROM (VALUES
  (1, 'ann', 10, 1.5), (2, 'bob', 5, NULL), (3, 'cy', 7, 2.0), (4, 'di', 1, 3.0), (6, 'fay', 3, 1.0), (NULL, 'nk', 1, 1.0)) t(id, name, qty, price);
CREATE TEMP TABLE v2 AS SELECT * FROM (VALUES
  (1, 'ann', 12, 1.5), (2, 'bob', 5, 2.5), (3, 'cyd', 8, 2.0), (5, 'ed', 2, 4.0), (6, 'fay', 3, 1.0), (NULL, 'nk', 1, 1.0)) t(id, name, qty, price);
-- the key must be unique on each side, or a record is compared more than once; a NULL key matches nothing, not even itself
CREATE TEMP TABLE repeated_keys AS
SELECT 'v1' AS side, id FROM v1 WHERE id IS NOT NULL GROUP BY id HAVING count(*) > 1
UNION ALL SELECT 'v2', id FROM v2 WHERE id IS NOT NULL GROUP BY id HAVING count(*) > 1;
-- every column as text, one row per key and column
CREATE TEMP TABLE old_cells AS
FROM (SELECT id, name::VARCHAR AS name, qty::VARCHAR AS qty, price::VARCHAR AS price FROM v1)
UNPIVOT INCLUDE NULLS (val FOR col IN (name, qty, price));
CREATE TEMP TABLE new_cells AS
FROM (SELECT id, name::VARCHAR AS name, qty::VARCHAR AS qty, price::VARCHAR AS price FROM v2)
UNPIVOT INCLUDE NULLS (val FOR col IN (name, qty, price));
-- cell by cell: one row for each cell that differs; IS DISTINCT FROM also sees NULL turning into a value
CREATE TEMP TABLE cell_report AS
SELECT id, col, o.val AS old_value, n.val AS new_value
FROM old_cells o JOIN new_cells n USING (id, col) WHERE o.val IS DISTINCT FROM n.val;
-- column by column: one row for each column compared
CREATE TEMP TABLE column_report AS
SELECT col, count(*) AS compared, count(*) FILTER (WHERE o.val IS DISTINCT FROM n.val) AS differing
FROM old_cells o JOIN new_cells n USING (id, col) GROUP BY col;
-- summary figures: one row for the two tables
CREATE TEMP TABLE summary AS
SELECT (SELECT count(*) FROM v1) AS rows_v1, (SELECT count(*) FROM v2) AS rows_v2,
       count(*) AS matched, count(*) FILTER (WHERE a IS DISTINCT FROM b) AS changed, count(*) FILTER (WHERE a IS NOT DISTINCT FROM b) AS identical,
       (SELECT count(*) FROM v1 ANTI JOIN v2 ON v1.id = v2.id) AS only_in_v1, (SELECT count(*) FROM v2 ANTI JOIN v1 ON v1.id = v2.id) AS only_in_v2
FROM v1 a JOIN v2 b ON a.id = b.id;
SELECT * FROM cell_report ORDER BY id, col;
SELECT * FROM column_report ORDER BY col;
SELECT * FROM summary;
SELECT CASE WHEN ((SELECT count(*) = 0 FROM repeated_keys)
  AND (SELECT list((id, col, old_value, new_value) ORDER BY id, col) IS NOT DISTINCT FROM [(1, 'qty', '10', '12'), (2, 'price', NULL, '2.5'), (3, 'name', 'cy', 'cyd'), (3, 'qty', '7', '8')] FROM cell_report)
  AND (SELECT map_from_entries(list((col, differing) ORDER BY col)) = MAP {'name': 1, 'price': 1, 'qty': 2} AND min(compared) = 4 AND max(compared) = 4 FROM column_report)
  AND (SELECT rows_v1 = 6 AND rows_v2 = 6 AND matched = 4 AND changed = 3 AND identical = 1 AND only_in_v1 = 2 AND only_in_v2 = 2 FROM summary)) THEN 'as expected' ELSE error('the report should hold 4 cells (qty of 1 and 3, the price of 2 turning from NULL to 2.5, the name of 3), 4 compared records in each column, and 4 matched records of which 3 changed, with ids 4 and 5 and the two NULL-key rows each unmatched') END AS verdict;
  • The statement builds its own data.
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the report should hold 4 cells (qty of 1 and 3, the price of 2 turning from NULL to 2.5, the name of 3), 4 compared records in each column, and 4 matched records of which 3 changed, with ids 4 and 5 and the two NULL-key rows each unmatched”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache Hop (1)
  • Table Compareread 2026-09-27

    Compare the contents of two database tables and report the differences

Bruin (1)
  • data-diffread 2026-09-27

    Compare two tables across connections: schema by default, per-column statistics with --full

Duckle (1)
EasyMorph (1)
  • Compare tablesread 2026-09-27

    Compare two tables and list the differences found per column

KNIME Analytics Platform (1)
  • Compare two tables cell by cell and by column spec, reporting the differences

Pentaho Data Integration (1)
  • Table Compareread 2026-09-27

    Compare the data of two same-layout tables and log the differences

VisiData (1)
  • setdiff-sheetread 2026-09-23

    (experimental) Set this sheet as the diff baseline; a colorizer marks cells that differ from it

Background

In pandas, DataFrame.compare compares two DataFrames element-wise and returns a DataFrame highlighting the differences.[1] KNIME's Table Difference Finder shows value differences for each row and column, and compares column types, domains and positions for each column.[2]

When to use it

EasyMorph's Compare tables action is for comparing the current version of a dataset against an earlier one to determine if it has changed, and what those changes were.[3] In EasyMorph, Compare tables returns an empty dataset when both tables are exactly the same.[3] Bruin's data-diff command is meant for comparing tables between development, staging and production, and for validating data migrations.[4]

In pandas, DataFrame.compare can only compare DataFrames with the same shape and identical row and column labels.[1] Apache Hop's Table Compare needs the two tables to have the same layout.[5] In Bruin, data-diff treats values whose percentage difference is below a tolerance, 0.001 by default, as equal.[4] Set a tolerance on purpose for floating-point columns, and report it with the result. (Meridian's view)

The method

SQLMesh's table diff runs a schema diff, which finds fields added, removed or changed in data type, and a row diff of data values.[6] SQLMesh's table diff reports how many rows are in both tables and in only one, and the percentage of matching values in each compared column of the joined rows.[6]

By default, Bruin's data-diff compares only schemas, and its --full flag adds row counts and per-column statistics such as null count, minimum and maximum.[4] EasyMorph's Compare tables reports differences per column, such as columns added or removed, values that have become or stopped being unique, and a changed number of rows.[3]

Apache Hop's Table Compare outputs a summary of the differences found, and each single difference through its error handling output.[5] In pandas, DataFrame.compare shows equal values as NaN unless keep_equal is set, and matching NaNs do not appear as a difference.[1]