Shape / inspect · All operations

Split valid from invalid rows

A variant of Check data quality

Send the rows that pass the rules to one output and the rows that fail to another, instead of stopping the run.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

The DuckDB step

Send the rows that pass the rules to one output and the rejects to another

It reads the orders table the proofs share. Create it first:

SQL
CREATE TABLE orders AS SELECT * FROM (VALUES
  (1,  'Alice Ng',      'north', DATE '2026-01-03', 120.00, 'widget', 'a|b',   '{"channel":"web","items":[1,2]}'),
  (2,  'alice ng',      'north', DATE '2026-01-05',  80.50, 'gadget', 'b',     '{"channel":"shop","items":[3]}'),
  (3,  'Bob  Okafor',   'south', DATE '2026-01-05', 300.00, 'widget', 'a|c',   '{"channel":"web","items":[4,5,6]}'),
  (4,  'Carla Díaz',    'east',  DATE '2026-01-09',  NULL,  'gizmo',  NULL,    '{"channel":"web","items":[]}'),
  (5,  'Bob Okafor',    'south', DATE '2026-01-12',  45.25, 'gadget', 'c',     '{"channel":"phone","items":[7]}'),
  (6,  'Dev Patel',     'west',  DATE '2026-01-12', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (7,  'Carla Diaz',    'east',  DATE '2026-01-20',  99.99, 'gizmo',  'b|c',   '{"channel":"shop","items":[10]}'),
  (8,  'Dev Patel',     'west',  DATE '2026-02-02', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (9,  'Eve Larsen',    NULL,    DATE '2026-02-03',  15.00, 'gadget', NULL,    '{"channel":"web","items":[11]}'),
  (10, 'Alice Ng',      'north', DATE '2026-02-10', 220.00, 'gizmo',  'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);

Then run the statement:

SQL
-- three rules; a NULL on a rule counts as a failure of it
CREATE TEMP TABLE checked AS
SELECT *,
       coalesce(amount IS NOT NULL, false) AS has_amount,
       coalesce(amount > 0, false) AS amount_positive,
       coalesce(region IS NOT NULL, false) AS has_region
FROM orders;
CREATE TEMP TABLE valid_rows AS
SELECT order_id, customer, region, amount FROM checked WHERE has_amount AND amount_positive AND has_region;
CREATE TEMP TABLE rejects AS
SELECT order_id, customer,
       concat_ws('; ', CASE WHEN NOT has_amount THEN 'amount missing' END,
                       CASE WHEN NOT amount_positive THEN 'amount not positive' END,
                       CASE WHEN NOT has_region THEN 'region missing' END) AS broken_rules
FROM checked WHERE NOT (has_amount AND amount_positive AND has_region);
SELECT * FROM valid_rows ORDER BY order_id;
SELECT * FROM rejects ORDER BY order_id;
SELECT CASE WHEN ((SELECT list(order_id ORDER BY order_id) FROM valid_rows) = [1, 2, 3, 5, 6, 7, 8, 10] AND (SELECT list((order_id, broken_rules) ORDER BY order_id) FROM rejects) = [(4, 'amount missing; amount not positive'), (9, 'region missing')] AND (SELECT count(*) FROM valid_rows) + (SELECT count(*) FROM rejects) = (SELECT count(*) FROM orders)) THEN 'as expected' ELSE error('the eight valid orders should go to one output and orders 4 and 9, with the rules each broke, to the other') END AS verdict;
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the eight valid orders should go to one output and orders 4 and 9, with the rules each broke, to the other”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache Hop (2)
  • Data Validatorread 2026-09-27

    Check fields against rules for data type, nulls, string length, min and max, allowed values

  • Check values against a target table and report the rows and columns an insert would reject

Apache NiFi (2)
  • ScriptedValidateRecordread 2026-09-27

    Route records as valid or invalid by the boolean a user script returns

  • ValidateRecordread 2026-09-27

    Validate records against a schema; valid and invalid records go to separate relationships

Apache SeaTunnel (1)
  • DataValidatorread 2026-09-27

    Validate fields by null, range, length and regex rules; fail, skip or route invalid rows to a table

Duckle (6)
  • Not-Null Checkread 2026-09-25

    pass rows with no nulls in chosen columns, the rest to reject

  • Range Checkread 2026-09-25

    pass in-range rows, the rest to reject

  • Referential Integrityread 2026-09-25

    pass rows whose key exists in a reference input, orphans to reject

  • Regex Matchread 2026-09-25

    pass rows matching a pattern, the rest to reject

  • Schema Validateread 2026-09-25

    reject rows where an expected column is null

  • Uniqueness Checkread 2026-09-25

    pass the first row per key, duplicates to reject

Pentaho Data Integration (1)
  • Data Validatorread 2026-09-27

    Validate passing rows against a set of rules

Talend (5)
  • 14 components checking table rows against data quality patterns or rules: tDBValidRows, tMySQLInvalidRows, tOracleValidRows

  • Validate rows against data quality rules held in Qlik Talend Data Integration or a Talend app

  • tMultiPatternCheckread 2026-09-27

    Check several columns against Java regular expressions, splitting matching from non-matching rows

  • tPatternCheckread 2026-09-27

    Check a column against a Java regular expression, splitting matching from non-matching rows

  • tSchemaComplianceCheckread 2026-09-27

    Validate rows against a reference schema: types, nullability and length

Background

Apache NiFi's ValidateRecord routes records that adhere to a schema to a valid relationship and records that do not to an invalid one.[1] In Apache NiFi, a single incoming file of records can be split into two outputs when some records are valid against the schema and others are not.[1]

When to use it

In PostgreSQL, a WHERE clause eliminates every row that does not satisfy its condition, and a row satisfies it only if the condition returns true.[2]

Apache Hop's Data Validator can report every validation error for a row, not only the first, and can concatenate the error codes and descriptions into one output row.[3] Carry the name of each broken rule on the reject row, so the rejects can be fixed and sent again without re-running every check. (Meridian's view)

When NiFi's ValidateRecord reads no records, as from a comma-separated values file with only a header line, it produces no output on any relationship and removes the incoming file.[1] An empty input then looks like a clean run; check for it before the split if an empty file should be noticed. (Meridian's view)

The method

NiFi's ValidateRecord can write invalid records with a separate writer, which helps when the main writer, as with Avro, cannot write data that does not adhere to its schema.[1] NiFi's ValidateRecord has an Allow Extra Fields property that decides whether a record carrying fields not present in the schema is still valid.[1]

In a query, compute each rule once as a true-or-false column, then write the rows where every rule holds to one table and the rest, with the rules they broke, to another. (Meridian's view) Keep the rejects in their original form where possible, so a fixed row can be sent back through the same checks. (Meridian's view)