Model / anomaly · All operations

Find outliers

Outlier or anomaly detection.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

Shape of the result
one row out per row in, with columns added or changed
Level
several clauses of SQL, or a window

The DuckDB step

Flag outliers by the interquartile rule

It reads the orders table the proofs share. Create it first:

SQL
CREATE TABLE orders AS SELECT * FROM (VALUES
  (1,  'Alice Ng',      'north', DATE '2026-01-03', 120.00, 'widget', 'a|b',   '{"channel":"web","items":[1,2]}'),
  (2,  'alice ng',      'north', DATE '2026-01-05',  80.50, 'gadget', 'b',     '{"channel":"shop","items":[3]}'),
  (3,  'Bob  Okafor',   'south', DATE '2026-01-05', 300.00, 'widget', 'a|c',   '{"channel":"web","items":[4,5,6]}'),
  (4,  'Carla Díaz',    'east',  DATE '2026-01-09',  NULL,  'gizmo',  NULL,    '{"channel":"web","items":[]}'),
  (5,  'Bob Okafor',    'south', DATE '2026-01-12',  45.25, 'gadget', 'c',     '{"channel":"phone","items":[7]}'),
  (6,  'Dev Patel',     'west',  DATE '2026-01-12', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (7,  'Carla Diaz',    'east',  DATE '2026-01-20',  99.99, 'gizmo',  'b|c',   '{"channel":"shop","items":[10]}'),
  (8,  'Dev Patel',     'west',  DATE '2026-02-02', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (9,  'Eve Larsen',    NULL,    DATE '2026-02-03',  15.00, 'gadget', NULL,    '{"channel":"web","items":[11]}'),
  (10, 'Alice Ng',      'north', DATE '2026-02-10', 220.00, 'gizmo',  'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);

Then run the statement:

SQL
CREATE TEMP TABLE result AS
WITH readings AS (SELECT order_id, amount FROM orders WHERE amount IS NOT NULL UNION ALL SELECT 99, 5000.0),
     q AS (SELECT quantile_cont(amount, 0.25) AS q1, quantile_cont(amount, 0.75) AS q3 FROM readings)
SELECT order_id, amount, amount > q3 + 1.5 * (q3 - q1) OR amount < q1 - 1.5 * (q3 - q1) AS outlier FROM readings, q;
SELECT * FROM result ORDER BY order_id;
SELECT CASE WHEN ((SELECT list(order_id) FILTER (WHERE outlier) = [99] AND count(*) = 10 FROM result)) THEN 'as expected' ELSE error('the one reading far from the rest was not the one flagged') END AS verdict;
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the one reading far from the rest was not the one flagged”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

AWS Glue (1)
  • DetectAnomaliesread 2026-09-27

    Detect anomalies in the metrics that another rule type reports over time

RapidMiner Studio (4)
  • Detect Outlier (COF)read 2026-09-27

    Flag outliers by Class Outlier Factor, which takes the class label into account

  • Detect Outlier (Densities)read 2026-09-27

    Flag rows as outliers when a given share of all rows lies farther than a distance D

  • Detect Outlier (Distances)read 2026-09-27

    Flag the n rows with the largest distance to their k-th nearest neighbour

  • Detect Outlier (LOF)read 2026-09-27

    Score rows by Local Outlier Factor from the density of their k nearest neighbours

Yggdrasil Decision Forests (2)
  • ANOMALY_DETECTIONread 2026-09-27

    Score each row between 0, most normal, and 1, most anomalous

  • ISOLATION_FORESTread 2026-09-27

    Train an isolation forest for anomaly detection; Python class IsolationForestLearner

Background

Finding outliers flags the observations that lie an abnormal distance from the other values in a sample, so that they can be investigated further.[1][2] Anomaly detection, also called outlier detection, identifies rare items, events or observations that deviate significantly from the majority of the data.[3] What counts as abnormal is left to the analyst, and normal observations must be characterised before abnormal ones can be singled out.[1]

When to use it

An outlier may indicate bad data, such as a value coded incorrectly or an experiment that was not run correctly.[2] Outliers may also be due to random variation, or may indicate something scientifically interesting.[2] An anomaly is not necessarily a data error: it may be a change in the process that generated the data, an unusual but valid observation, or an event such as fraud.[3] Outliers often contain valuable information about the process under investigation or about how the data was gathered and recorded.[1]

Removing a data point solely because it is an outlier is a controversial practice, as it typically invalidates statistical results.[4] Even when a normal distribution suits the data, outliers are expected in large samples and should not automatically be discarded.[4] Before eliminating outliers from the data, one should try to understand why they appeared and whether similar values are likely to continue to appear.[1] Flag outliers in a column of their own and keep the rows, so that any decision to drop one is made later, by a person, and can be seen. (Meridian's view)

Identifying an observation as an outlier depends on the underlying distribution of the data.[2] If data assumed to be approximately normal are not, an observation judged an outlier may reflect the non-normality of the data rather than an outlier.[2] Outliers can indicate that the population has a heavy-tailed distribution, in which case one should be very cautious with tools that assume a normal distribution.[4] A contextual anomaly is anomalous only within a particular context, as a temperature may be normal in summer but unusual in winter.[3]

The method

The interquartile range (IQR) is the difference between the upper quartile, Q3, and the lower quartile, Q1.[1][5] John Tukey proposed flagging any observation outside the range from Q1 minus k times the IQR to Q3 plus k times the IQR, a test known as Tukey's fences.[4] In Tukey's fences, a multiplier k of 1.5 indicates an outlier and a k of 3 indicates data that is far out.[4] A point beyond an inner fence, 1.5 times the IQR outside the quartiles, is a mild outlier, and a point beyond an outer fence, 3 times the IQR outside them, is an extreme outlier.[1]

The z-score of an observation is its distance from the sample mean in units of the sample standard deviation.[2] Using z-scores to identify possible outliers is common practice but can be misleading, particularly for small samples, because the maximum z-score is at most (n-1)/√n for a sample of n.[2] For normally distributed data, roughly 1 in 22 observations differs from the mean by twice the standard deviation or more, and 1 in 370 by three times.[4] Iglewicz and Hoaglin recommend a modified z-score built on the median and the median absolute deviation (MAD), labelling values above 3.5 in absolute value as potential outliers.[2] For data that follow an approximately lognormal distribution, taking logarithms transforms the data to normality before outlier tests for normal data are applied.[2]

Grubbs' test is the recommended test for a single outlier in approximately normal data.[2] The generalized extreme studentized deviate (ESD) test needs only an upper bound on the number of suspected outliers, and is recommended when the exact number is not known.[2] Masking can occur when a test looks for too few outliers, so that the additional outliers influence the test statistic enough that no point is declared an outlier.[2] Swamping can occur when a test looks for too many outliers, so that a single outlier and another point may both be declared outliers.[2] Applying a single-outlier test sequentially in order to detect multiple outliers is not appropriate.[2]

Many anomaly-detection methods return a numerical anomaly score rather than a label, and the score becomes a yes-or-no decision when a threshold is applied.[3] Unsupervised anomaly detection techniques assume the data is unlabelled, and are by far the most commonly used.[3] The local outlier factor (LOF) was proposed by Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng and Jörg Sander in 2000.[6] LOF compares the local density of a point, estimated from the distances to its k nearest neighbours, with the local densities of those neighbours.[6][7] Under LOF, points that have a substantially lower density than their neighbours are considered outliers.[6][7] Isolation forest, initially proposed by Liu et al. in 2008, isolates a point by recursively splitting the data on a random attribute at a random value between its minimum and maximum.[8] Isolation forest marks as anomalies the points with short average path lengths on its isolation trees, because anomalies need fewer random partitions to be isolated.[8] RapidMiner's Detect Outlier (Distances) operator follows Ramaswamy, Rastogi and Shim, ranking each point by its distance to its k-th nearest neighbour and declaring the top-ranked points outliers.[9]

Compared with the alternatives

The median is a robust statistic of central tendency, while the mean is not.[4] The IQR is used as a robust measure of scale, and has a breakdown point of 25%.[5] Robust statistical techniques are ones that will not be unduly affected by outliers.[2] A rule built on quartiles or the median is the better first choice for a single numeric column. (Meridian's view) Reach for a neighbourhood method such as LOF, or for isolation forest, when an odd row is odd only in the combination of its columns. (Meridian's view)

Sources

  1. [1]7.1.6. What are outliers in the data?secondary source
  2. [2]1.3.5.17. Detection of Outlierssecondary source
  3. [3]Anomaly detection - Wikipediasecondary source
  4. [4]Outlier - Wikipediasecondary source
  5. [5]Interquartile range - Wikipediasecondary source
  6. [6]Local outlier factor - Wikipediasecondary source
  7. [7]Detect Outlier (LOF) - Rapidminer Documentationdocumentation
  8. [8]Isolation forest - Wikipediasecondary source
  9. [9]Detect Outlier (Distances) - Rapidminer Documentationdocumentation