# Flag missing values

Shape / columns

A variant of [Fill in missing values](https://meridian.online/operations/fill-in-missing-values).

Add a column that marks whether each value is missing, leaving the value alone.

Page: <https://meridian.online/operations/fill-in-missing-values/flag-missing-values>

## Status

Runs as a SQL step.

arcform carries it as a `sql:` step today.

## The DuckDB step

add a column that flags missing values and leave the values alone

It reads the `orders` table the proofs share. Create it first:

```sql
CREATE TABLE orders AS SELECT * FROM (VALUES
  (1,  'Alice Ng',      'north', DATE '2026-01-03', 120.00, 'widget', 'a|b',   '{"channel":"web","items":[1,2]}'),
  (2,  'alice ng',      'north', DATE '2026-01-05',  80.50, 'gadget', 'b',     '{"channel":"shop","items":[3]}'),
  (3,  'Bob  Okafor',   'south', DATE '2026-01-05', 300.00, 'widget', 'a|c',   '{"channel":"web","items":[4,5,6]}'),
  (4,  'Carla Díaz',    'east',  DATE '2026-01-09',  NULL,  'gizmo',  NULL,    '{"channel":"web","items":[]}'),
  (5,  'Bob Okafor',    'south', DATE '2026-01-12',  45.25, 'gadget', 'c',     '{"channel":"phone","items":[7]}'),
  (6,  'Dev Patel',     'west',  DATE '2026-01-12', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (7,  'Carla Diaz',    'east',  DATE '2026-01-20',  99.99, 'gizmo',  'b|c',   '{"channel":"shop","items":[10]}'),
  (8,  'Dev Patel',     'west',  DATE '2026-02-02', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (9,  'Eve Larsen',    NULL,    DATE '2026-02-03',  15.00, 'gadget', NULL,    '{"channel":"web","items":[11]}'),
  (10, 'Alice Ng',      'north', DATE '2026-02-10', 220.00, 'gizmo',  'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);
```

Then run the statement:

```sql
CREATE TEMP TABLE result AS
SELECT *, amount IS NULL AS amount_missing, region IS NULL AS region_missing, tags IS NULL AS tags_missing FROM orders;
SELECT * FROM result ORDER BY order_id;
SELECT CASE WHEN ((SELECT list(column_name) FROM (DESCRIBE result)) = ['order_id', 'customer', 'region', 'order_date', 'amount', 'product', 'tags', 'meta', 'amount_missing', 'region_missing', 'tags_missing'] AND (SELECT count(*) FILTER (WHERE amount_missing) = 1 AND count(*) FILTER (WHERE region_missing) = 1 AND count(*) FILTER (WHERE tags_missing) = 2 AND count(*) = 10 AND count(amount) = 9 AND sum(amount) = 1900.74 FROM result)) THEN 'as expected' ELSE error('the missing amount, region and tags were not flagged in new columns with every value and every row kept') END AS verdict;
```

It needs nothing installed beyond DuckDB itself.

## What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

- Ibis: [`IsNull`](https://github.com/ibis-project/ibis/blob/05d2b293344e86cb41c1a426071734f7e270335a/ibis/expr/operations/generic.py) (read 2026-09-27)
- Ibis: [`NotNull`](https://github.com/ibis-project/ibis/blob/05d2b293344e86cb41c1a426071734f7e270335a/ibis/expr/operations/generic.py) (read 2026-09-27)
- KNIME Analytics Platform: [`Extract Missing Value Cause`](https://api.hub.knime.com/nodes/*WfViBjs3J9W6KB5x?details=full) (read 2026-09-27)
- OpenRefine: [`Facet by blank (null or empty string)`](https://openrefine.org/docs/manual/facets#facet-by-null-empty-or-blank) (read 2026-09-27)
- OpenRefine: [`Facet by empty string`](https://openrefine.org/docs/manual/facets#facet-by-null-empty-or-blank) (read 2026-09-27)
- OpenRefine: [`Facet by null`](https://openrefine.org/docs/manual/facets#facet-by-null-empty-or-blank) (read 2026-09-27)