# Split valid from invalid rows

Shape / inspect

A variant of [Check data quality](https://meridian.online/operations/check-data-quality).

Send the rows that pass the rules to one output and the rows that fail to another, instead of stopping the run.

Page: <https://meridian.online/operations/check-data-quality/split-valid-from-invalid-rows>

## Status

Runs as a SQL step.

arcform carries it as a `sql:` step today.

## The DuckDB step

send the rows that pass the rules to one output and the rejects to another

It reads the `orders` table the proofs share. Create it first:

```sql
CREATE TABLE orders AS SELECT * FROM (VALUES
  (1,  'Alice Ng',      'north', DATE '2026-01-03', 120.00, 'widget', 'a|b',   '{"channel":"web","items":[1,2]}'),
  (2,  'alice ng',      'north', DATE '2026-01-05',  80.50, 'gadget', 'b',     '{"channel":"shop","items":[3]}'),
  (3,  'Bob  Okafor',   'south', DATE '2026-01-05', 300.00, 'widget', 'a|c',   '{"channel":"web","items":[4,5,6]}'),
  (4,  'Carla Díaz',    'east',  DATE '2026-01-09',  NULL,  'gizmo',  NULL,    '{"channel":"web","items":[]}'),
  (5,  'Bob Okafor',    'south', DATE '2026-01-12',  45.25, 'gadget', 'c',     '{"channel":"phone","items":[7]}'),
  (6,  'Dev Patel',     'west',  DATE '2026-01-12', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (7,  'Carla Diaz',    'east',  DATE '2026-01-20',  99.99, 'gizmo',  'b|c',   '{"channel":"shop","items":[10]}'),
  (8,  'Dev Patel',     'west',  DATE '2026-02-02', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (9,  'Eve Larsen',    NULL,    DATE '2026-02-03',  15.00, 'gadget', NULL,    '{"channel":"web","items":[11]}'),
  (10, 'Alice Ng',      'north', DATE '2026-02-10', 220.00, 'gizmo',  'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);
```

Then run the statement:

```sql
-- three rules; a NULL on a rule counts as a failure of it
CREATE TEMP TABLE checked AS
SELECT *,
       coalesce(amount IS NOT NULL, false) AS has_amount,
       coalesce(amount > 0, false) AS amount_positive,
       coalesce(region IS NOT NULL, false) AS has_region
FROM orders;
CREATE TEMP TABLE valid_rows AS
SELECT order_id, customer, region, amount FROM checked WHERE has_amount AND amount_positive AND has_region;
CREATE TEMP TABLE rejects AS
SELECT order_id, customer,
       concat_ws('; ', CASE WHEN NOT has_amount THEN 'amount missing' END,
                       CASE WHEN NOT amount_positive THEN 'amount not positive' END,
                       CASE WHEN NOT has_region THEN 'region missing' END) AS broken_rules
FROM checked WHERE NOT (has_amount AND amount_positive AND has_region);
SELECT * FROM valid_rows ORDER BY order_id;
SELECT * FROM rejects ORDER BY order_id;
SELECT CASE WHEN ((SELECT list(order_id ORDER BY order_id) FROM valid_rows) = [1, 2, 3, 5, 6, 7, 8, 10] AND (SELECT list((order_id, broken_rules) ORDER BY order_id) FROM rejects) = [(4, 'amount missing; amount not positive'), (9, 'region missing')] AND (SELECT count(*) FROM valid_rows) + (SELECT count(*) FROM rejects) = (SELECT count(*) FROM orders)) THEN 'as expected' ELSE error('the eight valid orders should go to one output and orders 4 and 9, with the rules each broke, to the other') END AS verdict;
```

It needs nothing installed beyond DuckDB itself.

## What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

- Duckle: [`Not-Null Check`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/website/docs/components.html) (read 2026-09-25)
- Duckle: [`Range Check`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/website/docs/components.html) (read 2026-09-25)
- Duckle: [`Referential Integrity`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/frontend/src/workflow-ui/palette-data.ts) (read 2026-09-25)
- Duckle: [`Regex Match`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/website/docs/components.html) (read 2026-09-25)
- Duckle: [`Schema Validate`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/website/docs/components.html) (read 2026-09-25)
- Duckle: [`Uniqueness Check`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/website/docs/components.html) (read 2026-09-25)
- Apache Hop: [`Data Validator`](https://hop.apache.org/manual/latest/pipeline/transforms/validator.html) (read 2026-09-27)
- Apache Hop: [`Database value validation`](https://hop.apache.org/manual/next/pipeline/transforms/databasevaluevalidation.html) (read 2026-09-27)
- Apache NiFi: [`ScriptedValidateRecord`](https://nifi.apache.org/components/org.apache.nifi.processors.script.ScriptedValidateRecord/) (read 2026-09-27)
- Apache NiFi: [`ValidateRecord`](https://nifi.apache.org/components/org.apache.nifi.processors.standard.ValidateRecord/) (read 2026-09-27)
- Pentaho Data Integration: [`Data Validator`](https://docs.pentaho.com/pdia-data-integration/9.3-data-integration/pdi-transformation-steps-reference-overview) (read 2026-09-27)
- Apache SeaTunnel: [`DataValidator`](https://seatunnel.apache.org/docs/2.3.13/transforms/data-validator) (read 2026-09-27)
- Talend: [`Database valid and invalid rows components (family)`](https://help.qlik.com/talend/en-US/components/8.0/About) (read 2026-09-27)
- Talend: [`tDQRules/tDataQualityRules`](https://help.qlik.com/talend/en-US/components/8.0/validation/validation-component) (read 2026-09-27)
- Talend: [`tMultiPatternCheck`](https://help.qlik.com/talend/en-US/components/8.0/pattern-validation/pattern-validation-component) (read 2026-09-27)
- Talend: [`tPatternCheck`](https://help.qlik.com/talend/en-US/components/8.0/pattern-validation/pattern-validation-component) (read 2026-09-27)
- Talend: [`tSchemaComplianceCheck`](https://help.qlik.com/talend/en-US/components/8.0/validation/validation-component) (read 2026-09-27)