# Deduplicate rows Shape / rows distinct rows, unique by key, keep first or last per key Page: ## Status Runs as a SQL step. arcform carries it as a `sql:` step today. ## The DuckDB step keep the latest order per product and amount It reads the `orders` table the proofs share. Create it first: ```sql CREATE TABLE orders AS SELECT * FROM (VALUES (1, 'Alice Ng', 'north', DATE '2026-01-03', 120.00, 'widget', 'a|b', '{"channel":"web","items":[1,2]}'), (2, 'alice ng', 'north', DATE '2026-01-05', 80.50, 'gadget', 'b', '{"channel":"shop","items":[3]}'), (3, 'Bob Okafor', 'south', DATE '2026-01-05', 300.00, 'widget', 'a|c', '{"channel":"web","items":[4,5,6]}'), (4, 'Carla Díaz', 'east', DATE '2026-01-09', NULL, 'gizmo', NULL, '{"channel":"web","items":[]}'), (5, 'Bob Okafor', 'south', DATE '2026-01-12', 45.25, 'gadget', 'c', '{"channel":"phone","items":[7]}'), (6, 'Dev Patel', 'west', DATE '2026-01-12', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'), (7, 'Carla Diaz', 'east', DATE '2026-01-20', 99.99, 'gizmo', 'b|c', '{"channel":"shop","items":[10]}'), (8, 'Dev Patel', 'west', DATE '2026-02-02', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'), (9, 'Eve Larsen', NULL, DATE '2026-02-03', 15.00, 'gadget', NULL, '{"channel":"web","items":[11]}'), (10, 'Alice Ng', 'north', DATE '2026-02-10', 220.00, 'gizmo', 'a|b|c', '{"channel":"phone","items":[12,13]}') ) t(order_id, customer, region, order_date, amount, product, tags, meta); ``` Then run the statement: ```sql CREATE TEMP TABLE result AS SELECT DISTINCT ON (product, amount) * FROM orders ORDER BY product, amount, order_date DESC; SELECT * FROM result; SELECT CASE WHEN ((SELECT count(*) = 9 AND count(*) FILTER (WHERE order_id = 8) = 1 AND count(*) FILTER (WHERE order_id = 6) = 0 FROM result)) THEN 'as expected' ELSE error('the duplicate widget order was not reduced to the later one') END AS verdict; ``` It needs nothing installed beyond DuckDB itself. ## Variants - [Flag duplicates](https://meridian.online/operations/deduplicate-rows/flag-duplicates): Keep every row and add a column marking the rows that repeat another. - [Keep only the duplicates](https://meridian.online/operations/deduplicate-rows/keep-only-duplicates): Keep only the rows whose whole row or key appears more than once. - Keep one row per key: Drop repeated rows so one row survives for each distinct whole row or key, chosen by order where it matters. - [Separate unique rows from duplicates](https://meridian.online/operations/deduplicate-rows/separate-unique-rows-from-duplicates): Send the first row per key to one output and the repeats to a second output. ## What other tools call it Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2). - Alteryx Designer: [`Unique`](https://help.alteryx.com/current/en/designer/tools/preparation/unique-tool.html) (read 2026-09-27) - Bruin: [`bruin.deduplicate(relation, partition_by, order_by)`](https://getbruin.com/docs/bruin/assets/templating/macros.html) (read 2026-09-27) - Apache DataFusion: [`DISTINCT`](https://datafusion.apache.org/user-guide/sql/select.html) (read 2026-09-27) - Apache DataFusion: [`DISTINCT ON`](https://datafusion.apache.org/user-guide/sql/select.html) (read 2026-09-27) - Apache DataFusion: [`distinct`](https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html) (read 2026-09-27) - Apache DataFusion: [`distinct_on`](https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html) (read 2026-09-27) - dbt-utils: [`deduplicate`](https://raw.githubusercontent.com/dbt-labs/dbt-utils/main/README.md) (read 2026-09-27) - Duckle: [`Distinct`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/website/docs/components.html) (read 2026-09-25) - EasyMorph: [`Deduplicate rows`](https://help.easymorph.com/doku.php?id=transformations:deduplicate) (read 2026-09-27) - EasyMorph: [`Keep duplicates`](https://help.easymorph.com/doku.php?id=transformations:keepduplicates) (read 2026-09-27) - AWS Glue: [`Drop Duplicates`](https://docs.aws.amazon.com/glue/latest/dg/transforms-drop-duplicates.html) (read 2026-09-27) - AWS Glue: [`FlagDuplicateRows`](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-pyspark-transforms-FlagDuplicateRows.html) (read 2026-09-27) - AWS Glue: [`FlagDuplicatesInColumn`](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-pyspark-transforms-FlagDuplicatesInColumn.html) (read 2026-09-27) - AWS Glue: [`RemoveDuplicates`](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-pyspark-transforms-RemoveDuplicates.html) (read 2026-09-27) - Apache Hop: [`Unique Rows`](https://hop.apache.org/manual/latest/pipeline/transforms/uniquerows.html) (read 2026-09-27) - Apache Hop: [`Unique Rows (HashSet)`](https://hop.apache.org/manual/latest/pipeline/transforms/uniquerowsbyhashset.html) (read 2026-09-27) - Ibis: [`Distinct`](https://github.com/ibis-project/ibis/blob/05d2b293344e86cb41c1a426071734f7e270335a/ibis/expr/operations/relations.py) (read 2026-09-27) - KNIME Analytics Platform: [`Duplicate Row Filter`](https://api.hub.knime.com/nodes/*SQUECT-F0VhUjdbf?details=full) (read 2026-09-27) - Apache NiFi: [`DeduplicateRecord`](https://nifi.apache.org/components/org.apache.nifi.processors.standard.DeduplicateRecord/) (read 2026-09-27) - Apache NiFi: [`DetectDuplicate`](https://nifi.apache.org/components/org.apache.nifi.processors.standard.DetectDuplicate/) (read 2026-09-27) - OpenRefine: [`Remove duplicate rows`](https://github.com/OpenRefine/OpenRefine/blob/master/main/webapp/modules/core/scripts/views/data-table/data-table-view.js) (read 2026-09-27) - Pentaho Data Integration: [`Unique Rows`](https://docs.pentaho.com/pdia-data-integration/9.3-data-integration/pdi-transformation-steps-reference-overview) (read 2026-09-27) - Pentaho Data Integration: [`Unique Rows (HashSet)`](https://docs.pentaho.com/pdia-data-integration/9.3-data-integration/pdi-transformation-steps-reference-overview) (read 2026-09-27) - Polars: [`DataFrame.is_duplicated`](https://docs.pola.rs/api/python/stable/reference/dataframe/descriptive.html) (read 2026-09-27) - Polars: [`DataFrame.is_unique`](https://docs.pola.rs/api/python/stable/reference/dataframe/descriptive.html) (read 2026-09-27) - Polars: [`DataFrame.unique`](https://docs.pola.rs/api/python/stable/reference/dataframe/modify_select.html) (read 2026-09-27) - Polars: [`Expr.unique`](https://docs.pola.rs/api/python/stable/reference/expressions/computation.html) (read 2026-09-27) - Power Query M: [`Table.Distinct`](https://learn.microsoft.com/en-us/powerquery-m/table-functions#membership) (read 2026-09-27) - RapidMiner Studio: [`Remove Duplicates`](https://docs.rapidminer.com/latest/studio/operators/cleansing/duplicates/remove_duplicates.html) (read 2026-09-27) - RapidMiner Studio: [`Remove Duplicates (In Database)`](https://docs.rapidminer.com/latest/studio/operators/extensions/In-Database%20Processing/indb_cleansing/indb_distinct.html) (read 2026-09-27) - SQLMesh: [`@DEDUPLICATE`](https://github.com/SQLMesh/sqlmesh/blob/23b237cea7218cc8f338b89c14f0957394a0003b/docs/concepts/macros/sqlmesh_macros.md) (read 2026-09-27) - Tableau Prep: [`Identify Duplicate Rows`](https://help.tableau.com/current/prep/en-us/prep_remove_duplicates.htm) (read 2026-09-27) - Talend: [`tUniqRow`](https://help.qlik.com/talend/en-US/components/8.0/deduplication/deduplication-component) (read 2026-09-27) - Trifacta: [`Deduplicate Transform`](https://help.alteryx.com/aac/en/trifacta-classic/wrangle-language/language-appendices/transforms.html) (read 2026-09-27) - Trifacta: [`Remove duplicate rows`](https://help.alteryx.com/aac/en/trifacta-classic/reference/transformation-reference.html) (read 2026-09-27) - VisiData: [`dedupe-rows`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/features/dedupe.py#L113) (read 2026-09-23)