Shape / inspect · All operations
Compare two tables
A variant of Check data quality
Check that a table matches another or an expected result, row for row, by row count or by a summary figure.
Status
- In arcform
Runs as a SQL step
arcform carries it as a
sql:step today.
The DuckDB step
Compare two tables row for row, by row count and by a summary figure
It reads the orders table the proofs share. Create it first:
CREATE TABLE orders AS SELECT * FROM (VALUES
(1, 'Alice Ng', 'north', DATE '2026-01-03', 120.00, 'widget', 'a|b', '{"channel":"web","items":[1,2]}'),
(2, 'alice ng', 'north', DATE '2026-01-05', 80.50, 'gadget', 'b', '{"channel":"shop","items":[3]}'),
(3, 'Bob Okafor', 'south', DATE '2026-01-05', 300.00, 'widget', 'a|c', '{"channel":"web","items":[4,5,6]}'),
(4, 'Carla Díaz', 'east', DATE '2026-01-09', NULL, 'gizmo', NULL, '{"channel":"web","items":[]}'),
(5, 'Bob Okafor', 'south', DATE '2026-01-12', 45.25, 'gadget', 'c', '{"channel":"phone","items":[7]}'),
(6, 'Dev Patel', 'west', DATE '2026-01-12', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'),
(7, 'Carla Diaz', 'east', DATE '2026-01-20', 99.99, 'gizmo', 'b|c', '{"channel":"shop","items":[10]}'),
(8, 'Dev Patel', 'west', DATE '2026-02-02', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'),
(9, 'Eve Larsen', NULL, DATE '2026-02-03', 15.00, 'gadget', NULL, '{"channel":"web","items":[11]}'),
(10, 'Alice Ng', 'north', DATE '2026-02-10', 220.00, 'gizmo', 'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);Then run the statement:
-- expected: order 3 has a different amount, order 10 is missing and order 11 is extra, so it also holds 10 rows
CREATE TEMP TABLE expected AS
SELECT order_id, customer, amount FROM orders WHERE order_id <> 3 AND order_id <> 10
UNION ALL SELECT 3, 'Bob Okafor', 301.00
UNION ALL SELECT 11, 'Gus Hall', 12.00;
CREATE TEMP TABLE actual AS SELECT order_id, customer, amount FROM orders;
CREATE TEMP TABLE only_in_actual AS SELECT * FROM actual EXCEPT ALL SELECT * FROM expected;
CREATE TEMP TABLE only_in_expected AS SELECT * FROM expected EXCEPT ALL SELECT * FROM actual;
CREATE TEMP TABLE same_as_itself AS SELECT * FROM actual EXCEPT ALL SELECT * FROM actual;
CREATE TEMP TABLE figures AS
SELECT (SELECT count(*) FROM actual) AS actual_rows, (SELECT count(*) FROM expected) AS expected_rows,
(SELECT sum(amount) FROM actual) AS actual_total, (SELECT sum(amount) FROM expected) AS expected_total;
SELECT * FROM only_in_actual ORDER BY order_id;
SELECT * FROM only_in_expected ORDER BY order_id;
SELECT * FROM figures;
SELECT CASE WHEN ((SELECT list(order_id ORDER BY order_id) FROM only_in_actual) = [3, 10] AND (SELECT list(order_id ORDER BY order_id) FROM only_in_expected) = [3, 11] AND (SELECT count(*) FROM same_as_itself) = 0 AND (SELECT actual_rows = expected_rows FROM figures) AND (SELECT actual_total <> expected_total FROM figures) AND (SELECT expected_total - actual_total FROM figures) = -207.00) THEN 'as expected' ELSE error('the two tables should differ by orders 3 and 10 one way and 3 and 11 the other while holding the same number of rows, with different totals') END AS verdict;- It needs nothing installed beyond DuckDB itself.
- The statement checks its own result and raises an error when it is wrong: “the two tables should differ by orders 3 and 10 one way and 3 and 11 the other while holding the same number of rows, with different totals”.
- It passed on DuckDB
v2.0.0 (Cyanoptera) d54cbbe3df.
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
Alteryx Designer (1)
Expect Equalread 2026-09-27Assert that two data streams are identical and report an error when they are not
AWS Glue (2)
AggregateMatchread 2026-09-27Check that two datasets agree on summary metrics, such as a total
RowCountMatchread 2026-09-27Check that the row counts of two datasets match
Bruin (1)
Unit Testsread 2026-09-27Run the asset query against mock upstream rows as a read-only SELECT and compare with expected rows
dbt-utils (4)
cardinality_equalityread 2026-09-27Assert a column has the same value cardinality as a column in another model
equal_rowcountread 2026-09-27Assert two relations have the same number of rows, optionally per group
equalityread 2026-09-27Assert two relations hold the same rows, over all or chosen columns, with optional numeric precision
fewer_rows_thanread 2026-09-27Assert the model has fewer rows than a comparison relation, optionally per group
KNIME Analytics Platform (1)
KNIME Testing Framework UI (extension)read 2026-09-27Extension outside the base install: 29 current nodes: Block Programmatically (Table); Block Programmatically (Variable), …
Polars (1)
polars.testing asserts (family)read 2026-09-275 functions: assert_frame_equal, assert_frame_not_equal, assert_series_equal, assert_series_not_equal, assert_schema_equal
SQLMesh (1)
Unit testsread 2026-09-27YAML tests that run a model on given input rows and compare query or CTE output to expected rows
Background
A cross-system consistency check compares data held in different systems to make sure it is consistent.[1] A batch total checks for missing records by adding a numerical field across all the records in a batch and checking that the total is correct.[1] In DuckDB, EXCEPT selects the rows that occur only in the left query.[2]
When to use it
Systems may represent the same data differently, so comparing them can require a transformation, such as when one stores a name in one field and another in separate fields.[1]
In Amazon Web Services (AWS) Glue Data Quality, the AggregateMatch rule checks whether two datasets match by comparing summary metrics such as a total sales amount.[3] Treat a matching row count or total as a quick first signal, and compare row for row when the tables must agree exactly. (Meridian's view)
In DuckDB, JOIN, UNION and GROUP BY do not guarantee that row order is preserved.[4] Compare tables as collections of rows or by a key, never by position, unless an ORDER BY fixes the order on both sides. (Meridian's view) Polars' assert_frame_equal treats float values as equal when they are within a tolerance unless check_exact is set, with a default relative tolerance of 1e-05.[5] Decide how close two computed decimals must be to count as equal before comparing, and report that tolerance with the result. (Meridian's view)
In DuckDB, any comparison with a NULL value returns NULL, including NULL = NULL, while IS NOT DISTINCT FROM compares NULL values as equal to each other.[6] When two tables are matched on a key, decide how rows with a missing key should pair, and use IS NOT DISTINCT FROM if they should. (Meridian's view)
The method
In DuckDB, vanilla EXCEPT follows set semantics and eliminates duplicates, while EXCEPT ALL follows bag semantics.[2]
In DuckDB, traditional set operations combine queries by column position, require the same number of columns, and use the column names of the first query.[2]
Compared with the alternatives
In DuckDB, INTERSECT selects the rows that occur in the results of both queries.[2]
Sources
- [1]Data validation - Wikipediasecondary source
- [2]Set Operations – DuckDBdocumentation
- [3]DQDL rule type reference - AWS Gluedocumentation
- [4]Order Preservation – DuckDBdocumentation
- [5]polars.testing.assert_frame_equal — Polars documentationdocumentation
- [6]NULL Values – DuckDBdocumentation