Join / diff · All operations

Compare two tables

Compare two tables, report changed, added and removed rows.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

Shape of the result
one table from two or more
Level
several clauses of SQL, or a window

The DuckDB step

Compare two versions of a table by key

It reads the orders table the proofs share. Create it first:

SQL
CREATE TABLE orders AS SELECT * FROM (VALUES
  (1,  'Alice Ng',      'north', DATE '2026-01-03', 120.00, 'widget', 'a|b',   '{"channel":"web","items":[1,2]}'),
  (2,  'alice ng',      'north', DATE '2026-01-05',  80.50, 'gadget', 'b',     '{"channel":"shop","items":[3]}'),
  (3,  'Bob  Okafor',   'south', DATE '2026-01-05', 300.00, 'widget', 'a|c',   '{"channel":"web","items":[4,5,6]}'),
  (4,  'Carla Díaz',    'east',  DATE '2026-01-09',  NULL,  'gizmo',  NULL,    '{"channel":"web","items":[]}'),
  (5,  'Bob Okafor',    'south', DATE '2026-01-12',  45.25, 'gadget', 'c',     '{"channel":"phone","items":[7]}'),
  (6,  'Dev Patel',     'west',  DATE '2026-01-12', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (7,  'Carla Diaz',    'east',  DATE '2026-01-20',  99.99, 'gizmo',  'b|c',   '{"channel":"shop","items":[10]}'),
  (8,  'Dev Patel',     'west',  DATE '2026-02-02', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (9,  'Eve Larsen',    NULL,    DATE '2026-02-03',  15.00, 'gadget', NULL,    '{"channel":"web","items":[11]}'),
  (10, 'Alice Ng',      'north', DATE '2026-02-10', 220.00, 'gizmo',  'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);

Then run the statement:

SQL
CREATE TEMP TABLE result AS
WITH v2 AS (SELECT * REPLACE (CASE WHEN order_id = 3 THEN 310.0 ELSE amount END AS amount) FROM orders WHERE order_id <> 9
            UNION ALL SELECT 11, 'Gus Hale', 'north', DATE '2026-02-11', 5.0, 'gadget', NULL, '{}')
SELECT coalesce(a.order_id, b.order_id) AS order_id,
       CASE WHEN a.order_id IS NULL THEN 'added' WHEN b.order_id IS NULL THEN 'removed'
            WHEN a IS DISTINCT FROM b THEN 'changed' ELSE 'same' END AS status
FROM orders a FULL JOIN v2 b USING (order_id)
WHERE status <> 'same' ORDER BY 1;
SELECT * FROM result;
SELECT CASE WHEN ((SELECT map_from_entries(list((order_id, status) ORDER BY order_id)) = MAP {3: 'changed', 9: 'removed', 11: 'added'} FROM result)) THEN 'as expected' ELSE error('the difference is not order 3 changed, 9 removed and 11 added') END AS verdict;
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the difference is not order 3 changed, 9 removed and 11 added”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

Variants

Flag added, changed and removed rows

Match the rows of two tables on a key and tag each as unchanged, changed, added or removed.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Apache Hop (2)
  • Merge rows (diff)read 2026-09-27

    Compare a reference and a new sorted stream, flagging rows identical, changed, new or deleted

  • Table Compareread 2026-09-27

    Compare the contents of two database tables and report the differences

AWS Glue (1)
  • DatasetMatchread 2026-09-27

    Compare a dataset with a reference dataset and tell whether they are in sync

Bruin (1)
  • data-diffread 2026-09-27

    Compare two tables across connections: schema by default, per-column statistics with --full

Duckle (3)
  • Diff Detectread 2026-09-25

    tag inserted, updated and deleted rows against a previous snapshot

  • DuckLake Data Diffread 2026-09-25

    row-level change feed between two chosen snapshots of a DuckLake table

  • source against target reconciliation report joined on keys

EasyMorph (1)
  • Compare tablesread 2026-09-27

    Compare two tables and list the differences found per column

KNIME Analytics Platform (1)
  • Compare two tables cell by cell and by column spec, reporting the differences

Pentaho Data Integration (2)
  • Merge Rows (diff)read 2026-09-27

    Compare two sorted streams and flag rows equal, changed, deleted or new

  • Table Compareread 2026-09-27

    Compare the data of two same-layout tables and log the differences

Polars (1)
SQLMesh (1)
  • table_diffread 2026-09-27

    Schema diff and outer-join row diff of two tables, or one model across two environments

VisiData (1)
  • setdiff-sheetread 2026-09-23

    (experimental) Set this sheet as the diff baseline; a colorizer marks cells that differ from it

Background

In Apache Hop, Merge Rows (diff) matches previous and new rows on key fields and flags each as identical, changed, new or deleted.[1] In Apache Hop's Merge Rows (diff), a changed row is one whose key is found in both inputs but one or more of its compared values differ.[1] A full outer join combines the effect of left and right outer joins, giving NULL in every column of the table that lacks a matching row.[2]

When to use it

Apache Hop's Merge Rows (diff) is useful for comparing data collected at two different times.[1] In Apache Hop, when a source has no change log and no reliable timestamp or increasing ID, change data capture (CDC) is a comparison of two full extracts.[3] Reach for a full comparison when the question is what happened between two versions of a table, and a timestamp column cannot be trusted to say. (Meridian's view)

Apache Hop's Table Compare needs the key fields of the tables, because without them the two tables cannot be correctly joined.[4] In SQLMesh, the grain that joins the two tables in a table diff should be unique and not null, and a warning is shown when rows contain null or duplicate grains.[5]

In SQL, a comparison with NULL can never result in true or false, but always in a third logical result, Unknown.[6] In DuckDB, the = operator returns NULL when either input is NULL, while IS DISTINCT FROM treats NULL values as equal.[7]

The method

Apache Hop's Merge Rows (diff) is a streaming compare that does not sort, so both inputs must already be ordered on the keys to match.[3] In Apache Hop, if either stream into Merge Rows (diff) is out of order, the flags are wrong.[3] Apache Hop's Merge Rows (diff) walks both sorted streams once and keeps only the current pair of rows in memory.[3]

Compared with the alternatives

The SQL EXCEPT operator returns the distinct rows of one query that do not appear in a second result set.[8] In databases, change data capture (CDC) is a set of software design patterns used to determine and track the data that has changed.[9]