Model / timeseries · All operations
Measure autocorrelation or a spectrum
A variant of Resample a time series
Compute how a series relates to itself across lags, its frequency spectrum, or its load-cycle counts.
Status
- In arcform
Runs as a SQL step
arcform carries it as a
sql:step today.
The DuckDB step
Measure autocorrelation by lag, the frequency spectrum, and rainflow cycle counts
-- a series that repeats every four steps, twelve values long; the length is read from the series wherever it is needed
CREATE TEMP TABLE series AS SELECT t, [0, 1, 0, -1][t % 4 + 1]::DOUBLE AS x FROM range(12) s(t);
-- autocorrelation: how the series covaries with itself shifted by each lag, over its variance
CREATE TEMP TABLE acf AS
WITH m AS (SELECT avg(x) AS mu FROM series)
SELECT lag, sum((a.x - mu) * (b.x - mu)) / (SELECT sum(power(x - mu, 2)) FROM series, m) AS r
FROM range(0, 7) l(lag), series a JOIN series b ON b.t = a.t + lag, m GROUP BY lag;
-- spectrum: the periodogram at each frequency k in cycles per series length, the squared size of the discrete Fourier transform over n
CREATE TEMP TABLE spectrum AS
SELECT k, (power(sum(x * cos(2 * pi() * k * t / n)), 2) + power(sum(x * sin(2 * pi() * k * t / n)), 2)) / n AS power
FROM series, (SELECT count(*) AS n FROM series) c, range(0, 7) f(k) GROUP BY k, n;
-- load cycles: rainflow counting (ASTM E1049 three-point rule) over sequences of peaks and valleys; a full cycle counts 1, a half cycle 0.5.
-- Sequence 1 is the textbook one. Sequence 2 swings less each time, so nothing closes and all nine ranges are left as half cycles at the end.
CREATE TEMP TABLE reversals AS
SELECT s, generate_subscripts(v, 1) AS i, unnest(v) AS v
FROM (VALUES (1, [-2, 1, -3, 5, -1, 3, -4, 4, -2]), (2, [10, -9, 8, -7, 6, -5, 4, -3, 2, -1])) t(s, v);
CREATE TEMP TABLE walk AS
WITH RECURSIVE steps(s, step, next_i, stack, cycle_range, cycle_count) AS (
SELECT DISTINCT s, 0, 1, []::INTEGER[], NULL::INTEGER, NULL::DOUBLE FROM reversals
UNION ALL
SELECT s, step + 1,
CASE WHEN reduce THEN next_i ELSE next_i + 1 END,
CASE WHEN reduce AND len(stack) = 3 THEN stack[2:]
WHEN reduce THEN list_concat(stack[:len(stack) - 3], [stack[len(stack)]])
ELSE list_append(stack, (SELECT v FROM reversals r WHERE r.s = e.s AND r.i = e.next_i)) END,
CASE WHEN reduce THEN abs(stack[len(stack) - 1] - stack[len(stack) - 2]) END,
CASE WHEN reduce THEN CASE WHEN len(stack) = 3 THEN 0.5 ELSE 1.0 END END
FROM (SELECT *, len(stack) >= 3 AND abs(stack[len(stack)] - stack[len(stack) - 1]) >= abs(stack[len(stack) - 1] - stack[len(stack) - 2]) AS reduce FROM steps) e
WHERE reduce OR next_i <= (SELECT count(*) FROM reversals r WHERE r.s = e.s))
SELECT * FROM steps;
-- what the stack still holds at the end counts as half cycles, one per step between neighbours, however long the stack is
CREATE TEMP TABLE cycles AS
SELECT s, cycle_range AS range, cycle_count AS n FROM walk WHERE cycle_range IS NOT NULL
UNION ALL
SELECT s, unnest(list_transform(range(1, len(stack)), lambda j: abs(stack[j + 1] - stack[j]))), 0.5
FROM (SELECT s, stack FROM walk QUALIFY row_number() OVER (PARTITION BY s ORDER BY step DESC) = 1);
CREATE TEMP TABLE bins AS SELECT s, range, sum(n) AS cycles FROM cycles GROUP BY s, range;
SELECT * FROM acf ORDER BY lag;
SELECT * FROM spectrum ORDER BY k;
SELECT * FROM bins ORDER BY s, range;
-- the lag-4 autocorrelation is 2/3 and lag-2 is -5/6 (fewer pairs at longer lags); all the power sits at k = 3, the four-step cycle;
-- the textbook sequence gives one full cycle of range 4 and half cycles of ranges 3, 4, 6, 8, 8 and 9; the second gives nine half cycles of ranges 19, 17, ... 3
SELECT CASE WHEN ((SELECT list(round(r, 9) ORDER BY lag) = [1.0, 0.0, -0.833333333, 0.0, 0.666666667, 0.0, -0.5] FROM acf)
AND (SELECT list(round(power, 9) ORDER BY k) = [0.0, 0.0, 0.0, 3.0, 0.0, 0.0, 0.0] FROM spectrum)
AND (SELECT map_from_entries(list((range, cycles) ORDER BY range)) = MAP {3: 0.5, 4: 1.5, 6: 0.5, 8: 1.0, 9: 0.5} FROM bins WHERE s = 1)
AND (SELECT sum(cycles) = 4.0 FROM bins WHERE s = 1)
AND (SELECT list(range ORDER BY range DESC) = [19, 17, 15, 13, 11, 9, 7, 5, 3] AND min(cycles) = 0.5 AND max(cycles) = 0.5 FROM bins WHERE s = 2)) THEN 'as expected' ELSE error('the series should show autocorrelation of -5/6 at lag 2 and 2/3 at lag 4, all its power at frequency 3, the rainflow counts of the textbook sequence, and nine half cycles for the sequence that never closes') END AS verdict;- The statement builds its own data.
- It needs nothing installed beyond DuckDB itself.
- The statement checks its own result and raises an error when it is wrong: “the series should show autocorrelation of -5/6 at lag 2 and 2/3 at lag 4, all its power at frequency 3, the rainflow counts of the textbook sequence, and nine half cycles for the sequence that never closes”.
- It passed on DuckDB
v2.0.0 (Cyanoptera) d54cbbe3df.
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
RapidMiner Studio (3)
Autocorrelation / Autocovarianceread 2026-09-27Compute autocorrelation, partial autocorrelation or autocovariance of series columns by lag
Fast Fourier Transformationread 2026-09-27Compute the fast Fourier transform (amplitude and phase spectrum) of series columns
Rainflow Matrixread 2026-09-27Compute the rainflow-counting matrix of load cycles for a discretised series column
Background
Autocorrelation measures the correlation of a signal with a delayed copy of itself, quantifying the similarity between observations at different times.[1] A fast Fourier transform (FFT) computes the discrete Fourier transform (DFT), which decomposes a sequence of values into components of different frequencies.[2] The rainflow-counting algorithm converts a loading sequence of varying stress into constant amplitude stress reversals, to calculate fatigue life.[3]
When to use it
The autocorrelation function can be used to detect non-randomness in data and, if the data are not random, to identify an appropriate time series model.[4] When autocorrelation is used to detect non-randomness, usually only the lag 1 autocorrelation is of interest, while to identify a model it is usually plotted for many lags.[4] When data have a trend, autocorrelations at small lags tend to be large and positive, and when data are seasonal they are larger at multiples of the seasonal period.[5] Time series that show no autocorrelation are called white noise, and for white noise 95% of the autocorrelation spikes are expected to lie within ±1.96/√T, where T is the series length.[5]
In RapidMiner, the Fast Fourier Transformation operator transforms a series into the frequency space, which can identify prominent oscillations such as a possible seasonal component.[6] Rainflow cycles are used with Miner's rule to calculate fatigue damage, or in a crack growth equation to calculate crack increments, each giving an estimate of a component's fatigue life.[3] In RapidMiner, the Rainflow Matrix operator needs its series attribute discretised first, and relies on bin names following the same natural order as the values they represent.[7]
The method
The lag k autocorrelation sums the products of each value's deviation from the mean with the deviation k steps away, and divides by the sum of squared deviations.[4][5] The autocorrelation coefficients make up the autocorrelation function (ACF), and its plot is sometimes known as a correlogram.[5] In RapidMiner, the partial autocorrelation function correlates a series with its lagged values after regressing on the values at all shorter lags, and can help identify the extent of the lag in an autoregressive model.[8]
Computing the DFT directly from its definition takes O(n²) operations, and an FFT reduces that to O(n log n), where n is the length of the sequence.[2] James Cooley and John Tukey, generally credited with the modern generic FFT algorithm, published it in 1965, and Carl Friedrich Gauss's unpublished 1805 work used a very similar method.[2] The Wiener–Khinchin theorem relates the autocorrelation function to the power spectral density through the Fourier transform.[1] Using the Wiener–Khinchin theorem, an autocorrelation can be computed in order n log(n) with two fast Fourier transforms, where the brute force algorithm is order n².[1] In RapidMiner, the Fast Fourier Transformation operator returns the amplitude spectrum, optionally adds the phase spectrum, and takes its sample rate from the index attribute or from a parameter.[9]
Rainflow algorithms all start by eliminating the points that are not turning points, then find the closed cycles, and may be left with half-closed residual cycles at the end.[3]
Compared with the alternatives
Autocorrelation is a correlation coefficient between two values of the same variable at different times, rather than between two different variables.[4] In RapidMiner, autocovariance is the covariance of a series with itself and, unlike the autocorrelation function, it is not normed.[8]
Sources
- [1]Autocorrelation - Wikipediasecondary source
- [2]Fast Fourier transform - Wikipediasecondary source
- [3]Rainflow-counting algorithm - Wikipediasecondary source
- [4]1.3.5.12. Autocorrelationsecondary source
- [5]2 Time series graphics – Forecasting: Principles and Practicesecondary source
- [6]Classic Decomposition - Rapidminer Documentationdocumentation
- [7]Rainflow Matrix - Rapidminer Documentationdocumentation
- [8]Autocorrelation / Autocovariance - Rapidminer Documentationdocumentation
- [9]Fast Fourier Transformation - Rapidminer Documentationdocumentation