Shape / inspect · All operations

Check a table-wide measure

A variant of Check data quality

Compute one measure over the whole table, such as the row count, share of nulls, mean or age of the newest row, and check it against a threshold.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

The DuckDB step

Check row count, null share, mean and age of the newest row against thresholds

It reads the orders table the proofs share. Create it first:

SQL
CREATE TABLE orders AS SELECT * FROM (VALUES
  (1,  'Alice Ng',      'north', DATE '2026-01-03', 120.00, 'widget', 'a|b',   '{"channel":"web","items":[1,2]}'),
  (2,  'alice ng',      'north', DATE '2026-01-05',  80.50, 'gadget', 'b',     '{"channel":"shop","items":[3]}'),
  (3,  'Bob  Okafor',   'south', DATE '2026-01-05', 300.00, 'widget', 'a|c',   '{"channel":"web","items":[4,5,6]}'),
  (4,  'Carla Díaz',    'east',  DATE '2026-01-09',  NULL,  'gizmo',  NULL,    '{"channel":"web","items":[]}'),
  (5,  'Bob Okafor',    'south', DATE '2026-01-12',  45.25, 'gadget', 'c',     '{"channel":"phone","items":[7]}'),
  (6,  'Dev Patel',     'west',  DATE '2026-01-12', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (7,  'Carla Diaz',    'east',  DATE '2026-01-20',  99.99, 'gizmo',  'b|c',   '{"channel":"shop","items":[10]}'),
  (8,  'Dev Patel',     'west',  DATE '2026-02-02', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (9,  'Eve Larsen',    NULL,    DATE '2026-02-03',  15.00, 'gadget', NULL,    '{"channel":"web","items":[11]}'),
  (10, 'Alice Ng',      'north', DATE '2026-02-10', 220.00, 'gizmo',  'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);

Then run the statement:

SQL
-- the check is made as of a fixed date, so it gives the same answer on every day it is run
CREATE TEMP TABLE metrics AS
WITH m AS (
  SELECT count(*) AS row_count,
         1 - count(amount) / count(*) AS null_share,
         avg(amount) AS mean_amount,
         DATE '2026-02-15' - max(order_date) AS newest_age_days
  FROM orders)
SELECT 'at least 10 rows' AS rule, row_count AS measured, row_count >= 10 AS passed FROM m
UNION ALL SELECT 'no more than 20% of amounts missing', null_share, null_share <= 0.2 FROM m
UNION ALL SELECT 'mean amount between 200 and 210', mean_amount, mean_amount BETWEEN 200 AND 210 FROM m
UNION ALL SELECT 'newest order at most 7 days old', newest_age_days, newest_age_days <= 7 FROM m;
-- the same check as a gate: HAVING keeps the row only when the measure meets the threshold
CREATE TEMP TABLE gate_passes AS SELECT count(*) AS row_count FROM orders HAVING count(*) >= 10;
CREATE TEMP TABLE gate_fails AS SELECT count(*) AS row_count FROM orders HAVING count(*) >= 11;
SELECT rule, round(measured::DOUBLE, 4) AS measured, passed FROM metrics ORDER BY rule;
SELECT CASE WHEN ((SELECT map_from_entries(list((rule, passed) ORDER BY rule)) FROM metrics) = MAP {'at least 10 rows': true, 'mean amount between 200 and 210': false, 'newest order at most 7 days old': true, 'no more than 20% of amounts missing': true} AND (SELECT measured::DOUBLE FROM metrics WHERE rule LIKE 'newest%') = 5 AND (SELECT round(measured::DOUBLE, 4) FROM metrics WHERE rule LIKE 'mean%') = 211.1933 AND (SELECT round(measured::DOUBLE, 2) FROM metrics WHERE rule LIKE 'no more%') = 0.1 AND (SELECT count(*) FROM gate_passes) = 1 AND (SELECT count(*) FROM gate_fails) = 0) THEN 'as expected' ELSE error('the four table-wide measures should be 10 rows, 0.1 null share, a mean of 211.1933 (outside 200 to 210) and a newest row 5 days old, and HAVING should pass at 10 and fail at 11') END AS verdict;
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the four table-wide measures should be 10 rows, 0.1 null share, a mean of 211.1933 (outside 200 to 210) and a newest row 5 days old, and HAVING should pass at 10 and fail at 11”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache Hop (1)
AWS Glue (9)
  • Completenessread 2026-09-27

    Check the share of non-null, non-blank values in a column

  • DataFreshnessread 2026-09-27

    Check how recent the values of a date column are

  • DistinctValuesCountread 2026-09-27

    Check the number of distinct values in a column

  • Meanread 2026-09-27

    Check the mean of a column against a threshold

  • RowCountread 2026-09-27

    Check the number of rows against a threshold

  • StandardDeviationread 2026-09-27

    Check the standard deviation of a column against a threshold

  • Sumread 2026-09-27

    Check the sum of a column against a threshold

  • UniqueValueRatioread 2026-09-27

    Check the ratio of unique values in a column against a threshold

  • Uniquenessread 2026-09-27

    Check the share of values in a column that occur exactly once

dbt-utils (4)
  • at_least_oneread 2026-09-27

    Assert a column has at least one non-null value

  • not_constantread 2026-09-27

    Assert a column does not hold the same value in all rows

  • not_null_proportionread 2026-09-27

    Assert the share of non-null values in a column lies between at_least and at_most

  • recencyread 2026-09-27

    Assert a timestamp column holds data at least as recent as a given interval

Duckle (2)
  • check the age of the newest timestamp against a maximum

  • Run Baselineread 2026-09-25

    compare this run's profile with the median of previously accepted runs

SQLMesh (8)
  • at_least_oneread 2026-09-27

    Audit: Listed columns contain at least one non-NULL value

  • chi_squareread 2026-09-27

    Audit: Chi-square statistic for two categorical columns stays under a critical value

  • kl_divergenceread 2026-09-27

    Audit: Symmetrised Kullback-Leibler divergence between two columns stays under a threshold

  • mean_in_rangeread 2026-09-27

    Audit: A numeric column's mean lies in a range

  • not_constantread 2026-09-27

    Audit: Listed columns have at least two non-NULL values

  • not_null_proportionread 2026-09-27

    Audit: Proportion of NULL values in a column is no greater than a threshold

  • number_of_rowsread 2026-09-27

    Audit: Row count of the model exceeds a threshold

  • stddev_in_rangeread 2026-09-27

    Audit: A numeric column's standard deviation lies in a range

Background

In Amazon Web Services (AWS) Glue Data Quality, the RowCount rule checks whether a dataset's record count matches a threshold.[1] The dbt_utils recency test asserts that a timestamp column holds data at least as recent as a defined date interval.[2] The dbt_utils not_null_proportion test asserts that the proportion of non-null values in a column lies within a specified range.[2]

When to use it

In DuckDB, all general aggregate functions except count return NULL on empty groups.[3] Check the row count before any other measure, and treat a measure that comes back NULL as a failure rather than a pass. (Meridian's view)

In DuckDB, now() and current_timestamp return the date and time at the start of the current transaction.[4] Fix the as-of date in a freshness check that must give the same answer whenever it is re-run, and use the current time only where the check should answer for the moment it runs. (Meridian's view) In a timeliness check, timestamps captured as data is transformed may be compared against each other and against a defined service level agreement.[5]

AWS Glue Data Quality's DetectAnomalies rule checks for anomalies in the metrics that another rule type reports.[1] A fixed threshold suits a measure that should not move, such as a minimum row count; a measure that drifts with the business, such as a mean amount, is better judged against its own history. (Meridian's view)

The method

In DuckDB, count() returns the number of rows and count(arg) the number of rows where arg is not NULL.[3] Divide the second by the first for the share of a column that is filled, and compare that share with the threshold. (Meridian's view) In DuckDB, the HAVING clause filters after grouping, whereas the WHERE clause filters before it.[6]

Some dbt_utils tests accept a group_by_columns argument, since some checks can only be expressed, or are more precise, within a group.[2] A measure that passes for the whole table can hide a group that fails, such as one region with no recent rows; check per group where the groups matter. (Meridian's view)