Shape / inspect · All operations
Split valid from invalid rows
A variant of Check data quality
Send the rows that pass the rules to one output and the rows that fail to another, instead of stopping the run.
Status
- In arcform
Runs as a SQL step
arcform carries it as a
sql:step today.
The DuckDB step
Send the rows that pass the rules to one output and the rejects to another
It reads the orders table the proofs share. Create it first:
CREATE TABLE orders AS SELECT * FROM (VALUES
(1, 'Alice Ng', 'north', DATE '2026-01-03', 120.00, 'widget', 'a|b', '{"channel":"web","items":[1,2]}'),
(2, 'alice ng', 'north', DATE '2026-01-05', 80.50, 'gadget', 'b', '{"channel":"shop","items":[3]}'),
(3, 'Bob Okafor', 'south', DATE '2026-01-05', 300.00, 'widget', 'a|c', '{"channel":"web","items":[4,5,6]}'),
(4, 'Carla Díaz', 'east', DATE '2026-01-09', NULL, 'gizmo', NULL, '{"channel":"web","items":[]}'),
(5, 'Bob Okafor', 'south', DATE '2026-01-12', 45.25, 'gadget', 'c', '{"channel":"phone","items":[7]}'),
(6, 'Dev Patel', 'west', DATE '2026-01-12', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'),
(7, 'Carla Diaz', 'east', DATE '2026-01-20', 99.99, 'gizmo', 'b|c', '{"channel":"shop","items":[10]}'),
(8, 'Dev Patel', 'west', DATE '2026-02-02', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'),
(9, 'Eve Larsen', NULL, DATE '2026-02-03', 15.00, 'gadget', NULL, '{"channel":"web","items":[11]}'),
(10, 'Alice Ng', 'north', DATE '2026-02-10', 220.00, 'gizmo', 'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);Then run the statement:
-- three rules; a NULL on a rule counts as a failure of it
CREATE TEMP TABLE checked AS
SELECT *,
coalesce(amount IS NOT NULL, false) AS has_amount,
coalesce(amount > 0, false) AS amount_positive,
coalesce(region IS NOT NULL, false) AS has_region
FROM orders;
CREATE TEMP TABLE valid_rows AS
SELECT order_id, customer, region, amount FROM checked WHERE has_amount AND amount_positive AND has_region;
CREATE TEMP TABLE rejects AS
SELECT order_id, customer,
concat_ws('; ', CASE WHEN NOT has_amount THEN 'amount missing' END,
CASE WHEN NOT amount_positive THEN 'amount not positive' END,
CASE WHEN NOT has_region THEN 'region missing' END) AS broken_rules
FROM checked WHERE NOT (has_amount AND amount_positive AND has_region);
SELECT * FROM valid_rows ORDER BY order_id;
SELECT * FROM rejects ORDER BY order_id;
SELECT CASE WHEN ((SELECT list(order_id ORDER BY order_id) FROM valid_rows) = [1, 2, 3, 5, 6, 7, 8, 10] AND (SELECT list((order_id, broken_rules) ORDER BY order_id) FROM rejects) = [(4, 'amount missing; amount not positive'), (9, 'region missing')] AND (SELECT count(*) FROM valid_rows) + (SELECT count(*) FROM rejects) = (SELECT count(*) FROM orders)) THEN 'as expected' ELSE error('the eight valid orders should go to one output and orders 4 and 9, with the rules each broke, to the other') END AS verdict;- It needs nothing installed beyond DuckDB itself.
- The statement checks its own result and raises an error when it is wrong: “the eight valid orders should go to one output and orders 4 and 9, with the rules each broke, to the other”.
- It passed on DuckDB
v2.0.0 (Cyanoptera) d54cbbe3df.
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
Apache Hop (2)
Data Validatorread 2026-09-27Check fields against rules for data type, nulls, string length, min and max, allowed values
Database value validationread 2026-09-27Check values against a target table and report the rows and columns an insert would reject
Apache NiFi (2)
ScriptedValidateRecordread 2026-09-27Route records as valid or invalid by the boolean a user script returns
ValidateRecordread 2026-09-27Validate records against a schema; valid and invalid records go to separate relationships
Apache SeaTunnel (1)
DataValidatorread 2026-09-27Validate fields by null, range, length and regex rules; fail, skip or route invalid rows to a table
Duckle (6)
Not-Null Checkread 2026-09-25pass rows with no nulls in chosen columns, the rest to reject
Range Checkread 2026-09-25pass in-range rows, the rest to reject
Referential Integrityread 2026-09-25pass rows whose key exists in a reference input, orphans to reject
Regex Matchread 2026-09-25pass rows matching a pattern, the rest to reject
Schema Validateread 2026-09-25reject rows where an expected column is null
Uniqueness Checkread 2026-09-25pass the first row per key, duplicates to reject
Pentaho Data Integration (1)
Data Validatorread 2026-09-27Validate passing rows against a set of rules
Talend (5)
Database valid and invalid rows components (family)read 2026-09-2714 components checking table rows against data quality patterns or rules: tDBValidRows, tMySQLInvalidRows, tOracleValidRows
tDQRules/tDataQualityRulesread 2026-09-27Validate rows against data quality rules held in Qlik Talend Data Integration or a Talend app
tMultiPatternCheckread 2026-09-27Check several columns against Java regular expressions, splitting matching from non-matching rows
tPatternCheckread 2026-09-27Check a column against a Java regular expression, splitting matching from non-matching rows
tSchemaComplianceCheckread 2026-09-27Validate rows against a reference schema: types, nullability and length
Background
Apache NiFi's ValidateRecord routes records that adhere to a schema to a valid relationship and records that do not to an invalid one.[1] In Apache NiFi, a single incoming file of records can be split into two outputs when some records are valid against the schema and others are not.[1]
When to use it
In PostgreSQL, a WHERE clause eliminates every row that does not satisfy its condition, and a row satisfies it only if the condition returns true.[2]
Apache Hop's Data Validator can report every validation error for a row, not only the first, and can concatenate the error codes and descriptions into one output row.[3] Carry the name of each broken rule on the reject row, so the rejects can be fixed and sent again without re-running every check. (Meridian's view)
When NiFi's ValidateRecord reads no records, as from a comma-separated values file with only a header line, it produces no output on any relationship and removes the incoming file.[1] An empty input then looks like a clean run; check for it before the split if an empty file should be noticed. (Meridian's view)
The method
NiFi's ValidateRecord can write invalid records with a separate writer, which helps when the main writer, as with Avro, cannot write data that does not adhere to its schema.[1] NiFi's ValidateRecord has an Allow Extra Fields property that decides whether a record carrying fields not present in the schema is still valid.[1]
In a query, compute each rule once as a true-or-false column, then write the rows where every rule holds to one table and the rest, with the rules they broke, to another. (Meridian's view) Keep the rejects in their original form where possible, so a fixed row can be sent back through the same checks. (Meridian's view)
Sources
- [1]ValidateRecord - Apache NiFidocumentation
- [2]PostgreSQL: Documentation: 18: SELECTdocumentation
- [3]Data Validator | Apache Hopdocumentation