Shape / inspect · All operations
Check a table-wide measure
A variant of Check data quality
Compute one measure over the whole table, such as the row count, share of nulls, mean or age of the newest row, and check it against a threshold.
Status
- In arcform
Runs as a SQL step
arcform carries it as a
sql:step today.
The DuckDB step
Check row count, null share, mean and age of the newest row against thresholds
It reads the orders table the proofs share. Create it first:
CREATE TABLE orders AS SELECT * FROM (VALUES
(1, 'Alice Ng', 'north', DATE '2026-01-03', 120.00, 'widget', 'a|b', '{"channel":"web","items":[1,2]}'),
(2, 'alice ng', 'north', DATE '2026-01-05', 80.50, 'gadget', 'b', '{"channel":"shop","items":[3]}'),
(3, 'Bob Okafor', 'south', DATE '2026-01-05', 300.00, 'widget', 'a|c', '{"channel":"web","items":[4,5,6]}'),
(4, 'Carla Díaz', 'east', DATE '2026-01-09', NULL, 'gizmo', NULL, '{"channel":"web","items":[]}'),
(5, 'Bob Okafor', 'south', DATE '2026-01-12', 45.25, 'gadget', 'c', '{"channel":"phone","items":[7]}'),
(6, 'Dev Patel', 'west', DATE '2026-01-12', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'),
(7, 'Carla Diaz', 'east', DATE '2026-01-20', 99.99, 'gizmo', 'b|c', '{"channel":"shop","items":[10]}'),
(8, 'Dev Patel', 'west', DATE '2026-02-02', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'),
(9, 'Eve Larsen', NULL, DATE '2026-02-03', 15.00, 'gadget', NULL, '{"channel":"web","items":[11]}'),
(10, 'Alice Ng', 'north', DATE '2026-02-10', 220.00, 'gizmo', 'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);Then run the statement:
-- the check is made as of a fixed date, so it gives the same answer on every day it is run
CREATE TEMP TABLE metrics AS
WITH m AS (
SELECT count(*) AS row_count,
1 - count(amount) / count(*) AS null_share,
avg(amount) AS mean_amount,
DATE '2026-02-15' - max(order_date) AS newest_age_days
FROM orders)
SELECT 'at least 10 rows' AS rule, row_count AS measured, row_count >= 10 AS passed FROM m
UNION ALL SELECT 'no more than 20% of amounts missing', null_share, null_share <= 0.2 FROM m
UNION ALL SELECT 'mean amount between 200 and 210', mean_amount, mean_amount BETWEEN 200 AND 210 FROM m
UNION ALL SELECT 'newest order at most 7 days old', newest_age_days, newest_age_days <= 7 FROM m;
-- the same check as a gate: HAVING keeps the row only when the measure meets the threshold
CREATE TEMP TABLE gate_passes AS SELECT count(*) AS row_count FROM orders HAVING count(*) >= 10;
CREATE TEMP TABLE gate_fails AS SELECT count(*) AS row_count FROM orders HAVING count(*) >= 11;
SELECT rule, round(measured::DOUBLE, 4) AS measured, passed FROM metrics ORDER BY rule;
SELECT CASE WHEN ((SELECT map_from_entries(list((rule, passed) ORDER BY rule)) FROM metrics) = MAP {'at least 10 rows': true, 'mean amount between 200 and 210': false, 'newest order at most 7 days old': true, 'no more than 20% of amounts missing': true} AND (SELECT measured::DOUBLE FROM metrics WHERE rule LIKE 'newest%') = 5 AND (SELECT round(measured::DOUBLE, 4) FROM metrics WHERE rule LIKE 'mean%') = 211.1933 AND (SELECT round(measured::DOUBLE, 2) FROM metrics WHERE rule LIKE 'no more%') = 0.1 AND (SELECT count(*) FROM gate_passes) = 1 AND (SELECT count(*) FROM gate_fails) = 0) THEN 'as expected' ELSE error('the four table-wide measures should be 10 rows, 0.1 null share, a mean of 211.1933 (outside 200 to 210) and a newest row 5 days old, and HAVING should pass at 10 and fail at 11') END AS verdict;- It needs nothing installed beyond DuckDB itself.
- The statement checks its own result and raises an error when it is wrong: “the four table-wide measures should be 10 rows, 0.1 null share, a mean of 211.1933 (outside 200 to 210) and a newest row 5 days old, and HAVING should pass at 10 and fail at 11”.
- It passed on DuckDB
v2.0.0 (Cyanoptera) d54cbbe3df.
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
Apache Hop (1)
Evaluate rows number in a tableread 2026-09-27Test the row count of a table or query against a condition to branch the workflow
AWS Glue (9)
Completenessread 2026-09-27Check the share of non-null, non-blank values in a column
DataFreshnessread 2026-09-27Check how recent the values of a date column are
DistinctValuesCountread 2026-09-27Check the number of distinct values in a column
Meanread 2026-09-27Check the mean of a column against a threshold
RowCountread 2026-09-27Check the number of rows against a threshold
StandardDeviationread 2026-09-27Check the standard deviation of a column against a threshold
Sumread 2026-09-27Check the sum of a column against a threshold
UniqueValueRatioread 2026-09-27Check the ratio of unique values in a column against a threshold
Uniquenessread 2026-09-27Check the share of values in a column that occur exactly once
dbt-utils (4)
at_least_oneread 2026-09-27Assert a column has at least one non-null value
not_constantread 2026-09-27Assert a column does not hold the same value in all rows
not_null_proportionread 2026-09-27Assert the share of non-null values in a column lies between at_least and at_most
recencyread 2026-09-27Assert a timestamp column holds data at least as recent as a given interval
Duckle (2)
Freshness Check (SLA Gate)read 2026-09-25check the age of the newest timestamp against a maximum
Run Baselineread 2026-09-25compare this run's profile with the median of previously accepted runs
SQLMesh (8)
at_least_oneread 2026-09-27Audit: Listed columns contain at least one non-NULL value
chi_squareread 2026-09-27Audit: Chi-square statistic for two categorical columns stays under a critical value
kl_divergenceread 2026-09-27Audit: Symmetrised Kullback-Leibler divergence between two columns stays under a threshold
mean_in_rangeread 2026-09-27Audit: A numeric column's mean lies in a range
not_constantread 2026-09-27Audit: Listed columns have at least two non-NULL values
not_null_proportionread 2026-09-27Audit: Proportion of NULL values in a column is no greater than a threshold
number_of_rowsread 2026-09-27Audit: Row count of the model exceeds a threshold
stddev_in_rangeread 2026-09-27Audit: A numeric column's standard deviation lies in a range
Background
In Amazon Web Services (AWS) Glue Data Quality, the RowCount rule checks whether a dataset's record count matches a threshold.[1] The dbt_utils recency test asserts that a timestamp column holds data at least as recent as a defined date interval.[2] The dbt_utils not_null_proportion test asserts that the proportion of non-null values in a column lies within a specified range.[2]
When to use it
In DuckDB, all general aggregate functions except count return NULL on empty groups.[3] Check the row count before any other measure, and treat a measure that comes back NULL as a failure rather than a pass. (Meridian's view)
In DuckDB, now() and current_timestamp return the date and time at the start of the current transaction.[4] Fix the as-of date in a freshness check that must give the same answer whenever it is re-run, and use the current time only where the check should answer for the moment it runs. (Meridian's view) In a timeliness check, timestamps captured as data is transformed may be compared against each other and against a defined service level agreement.[5]
AWS Glue Data Quality's DetectAnomalies rule checks for anomalies in the metrics that another rule type reports.[1] A fixed threshold suits a measure that should not move, such as a minimum row count; a measure that drifts with the business, such as a mean amount, is better judged against its own history. (Meridian's view)
The method
In DuckDB, count() returns the number of rows and count(arg) the number of rows where arg is not NULL.[3] Divide the second by the first for the share of a column that is filled, and compare that share with the threshold. (Meridian's view) In DuckDB, the HAVING clause filters after grouping, whereas the WHERE clause filters before it.[6]
Some dbt_utils tests accept a group_by_columns argument, since some checks can only be expressed, or are more precise, within a group.[2] A measure that passes for the whole table can hide a group that fails, such as one region with no recent rows; check per group where the groups matter. (Meridian's view)
Sources
- [1]DQDL rule type reference - AWS Gluedocumentation
- [2]dbt-utils README.mddocumentation
- [3]Aggregate Functions – DuckDBdocumentation
- [4]Timestamp with Time Zone Functions – DuckDBdocumentation
- [5]Data quality - Wikipediasecondary source
- [6]HAVING Clause – DuckDBdocumentation