Model / evaluate · All operations

Cross-validate a model

A variant of Evaluate a model

Train and test a model repeatedly over folds, hold-out splits or bootstrap samples and pool the scores.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

The DuckDB step

Cross-validate a line by folds, by hold-out splits and by bootstrap samples, and pool the errors

SQL
-- twelve observations of a noisy line, and the model to validate: a least-squares line fitted with regr_slope and regr_intercept
CREATE TEMP TABLE data AS SELECT i AS id, i::DOUBLE AS x, y FROM (VALUES
  (1, 3.1), (2, 4.8), (3, 7.3), (4, 8.9), (5, 11.2), (6, 12.8), (7, 15.1), (8, 17.3), (9, 18.7), (10, 21.2), (11, 22.8), (12, 25.1)) t(i, y);
-- a bootstrap draw takes twelve rows with replacement; the draws are a fixed formula so that the proof gives the same samples on every run
CREATE TEMP TABLE draws AS SELECT b, (j * j + 3 * b * j + b) % 13 % 12 + 1 AS id FROM range(1, 4) rb(b), range(12) rj(j);
-- one table says, for every scheme and split, which rows train the model and which test it (a drawn row trains once per draw)
CREATE TEMP TABLE splits AS
SELECT 'folds' AS scheme, f AS split, id, CASE WHEN (id - 1) % 3 = f THEN 'test' ELSE 'train' END AS role FROM data, range(3) r(f)
UNION ALL
SELECT 'hold-outs', s, id, CASE WHEN id % 4 = s THEN 'test' ELSE 'train' END FROM data, range(2) r(s)
UNION ALL
SELECT 'bootstrap', b, id, 'train' FROM draws
UNION ALL
SELECT 'bootstrap', b, id, 'test' FROM data, range(1, 4) rb(b) WHERE NOT EXISTS (SELECT 1 FROM draws d WHERE d.b = rb.b AND d.id = data.id);
CREATE TEMP TABLE fits AS
SELECT scheme, split, regr_slope(y, x) AS slope, regr_intercept(y, x) AS intercept
FROM splits JOIN data USING (id) WHERE role = 'train' GROUP BY scheme, split;
CREATE TEMP TABLE errors AS
SELECT scheme, split, y - (intercept + slope * x) AS err
FROM splits JOIN data USING (id) JOIN fits USING (scheme, split) WHERE role = 'test';
CREATE TEMP TABLE per_split AS SELECT scheme, split, count(*) AS n_test, sqrt(avg(err * err)) AS rmse FROM errors GROUP BY scheme, split;
-- pool: the root mean square of every held-out error of the scheme, not an average of the splits' scores
CREATE TEMP TABLE result AS
SELECT scheme, count(*) AS splits, sum(n_test) AS n_test, (SELECT sqrt(avg(err * err)) FROM errors e WHERE e.scheme = p.scheme) AS pooled_rmse
FROM per_split p GROUP BY scheme;
SELECT scheme, splits, n_test, round(pooled_rmse, 6) AS pooled_rmse FROM result ORDER BY scheme;
-- the expected values were computed outside DuckDB with the same splits: folds test every row once (12), the two hold-outs test 3 rows each,
-- and the bootstrap tests the 5, 5 and 6 rows each sample left out; the tolerance 1e-8 is for float rounding only
SELECT CASE WHEN ((SELECT map_from_entries(list((scheme, (splits, n_test)) ORDER BY scheme)) = MAP {'bootstrap': (3, 16), 'folds': (3, 12), 'hold-outs': (2, 6)} FROM result)
  AND (SELECT abs(pooled_rmse - 0.218900778) < 1e-8 FROM result WHERE scheme = 'folds')
  AND (SELECT abs(pooled_rmse - 0.239645227) < 1e-8 FROM result WHERE scheme = 'hold-outs')
  AND (SELECT abs(pooled_rmse - 0.23247675) < 1e-8 FROM result WHERE scheme = 'bootstrap')
  AND (SELECT list(round(rmse, 6) ORDER BY split) = [0.143963, 0.230184, 0.264655] FROM per_split WHERE scheme = 'folds')
  AND (SELECT count(*) = 12 AND count(DISTINCT id) = 12 FROM splits WHERE scheme = 'folds' AND role = 'test')) THEN 'as expected' ELSE error('the pooled held-out errors of the folds, hold-outs and bootstrap samples should be 0.2189, 0.2396 and 0.2325, with every row tested once in the folds') END AS verdict;
  • The statement builds its own data.
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the pooled held-out errors of the folds, hold-outs and bootstrap samples should be 0.2189, 0.2396 and 0.2325, with every row tested once in the folds”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Alteryx Designer (1)
  • Cross-Validationread 2026-09-27

    Compare one or more predictive models by k-fold cross-validation

KNIME Analytics Platform (2)
  • X-Aggregatorread 2026-09-27

    Close a cross-validation loop, collecting predictions and error rates per fold

  • X-Partitionerread 2026-09-27

    Open a cross-validation loop, splitting rows into training and test folds per iteration

mlpack (1)
  • Cross-validationread 2026-09-27

    KFoldCV and SimpleCV: score any mlpack method over k folds or one hold-out split

RapidMiner Studio (5)
  • Bootstrapping Validationread 2026-09-27

    Estimate model performance by training on bootstrap samples and testing on the rest

  • Cross Validationread 2026-09-27

    k-fold cross validation of an inner training and testing process

  • Split Validationread 2026-09-27

    Hold-out validation: split into training and test sets, train, then test

  • Wrapper Split Validationread 2026-09-27

    Hold-out validation of a feature weighting or selection wrapper

  • Wrapper-X-Validationread 2026-09-27

    Cross validation of a feature weighting or selection wrapper

Yggdrasil Decision Forests (1)
  • cross_validationread 2026-09-27

    Train and score the learner over k folds and return one pooled evaluation

Background

Cross-validation assesses how a statistical analysis will generalise to an independent data set, using different portions of the data to train and test.[1] In k-fold cross-validation the sample is split into k equal folds, and each fold is the validation data exactly once while the other k − 1 train the model.[1] The k results of k-fold cross-validation can then be averaged to produce a single estimate.[1]

When to use it

The goal of cross-validation is to test a model's ability to predict new data that was not used in estimating it, to flag problems like overfitting or selection bias.[1] The holdout method, a single split into a training set and a test set, should be used with caution, because without averaging over several runs it can give highly misleading results.[1] Cross-validation only yields meaningful results if the validation set and training set are drawn from the same population.[1]

Scikit-learn's KFold and ShuffleSplit splitters assume samples are independent and identically distributed, and on time series data they give poor estimates of generalisation error.[2] Scikit-learn's TimeSeriesSplit instead trains on the first k folds and tests on the fold after them, so successive training sets are supersets of those before.[2] Where samples come in groups, such as several samples from each subject, scikit-learn's GroupKFold ensures the same group is not in both the testing and the training sets.[2] In scikit-learn, splitting data with a rare class can give training or validation folds with no example of that class, which typically leaves classification metrics undefined.[2]

If feature selection or model tuning is part of the modelling procedure, it must be repeated on every training set, or the predictions will be upwardly biased.[1] Mean-centring, rescaling, outlier removal or other data-dependent preprocessing done on the entire data set has been shown to bias cross-validation estimates.[1] Fit every step that learns from the data, scaling and feature selection included, inside each fold, never once on the whole table. (Meridian's view)

The method

In stratified k-fold cross-validation the folds are chosen so that, for a binary classification, each holds roughly the same proportions of the two class labels.[1] 10-fold cross-validation is commonly used, but in general k remains an unfixed parameter.[1] When k equals the number of observations, k-fold cross-validation is equivalent to leave-one-out cross-validation.[1] Leave-p-out cross-validation trains and validates on every way of choosing p observations for validation, and for p above 1 and even moderately large samples it can become computationally infeasible.[1]

Repeated random sub-sampling, also known as Monte Carlo cross-validation, averages over many random splits, so some observations may never be validated while others are validated more than once.[1] RapidMiner's Bootstrapping Validation splits the rows into a training and a test subset, draws a sample with replacement from the training subset, and repeats the round as often as its number of validations parameter says.[3]

When cross-validation is used both to choose a model's settings, its hyperparameters, and to estimate its error, nested cross-validation is required.[1] In the k*l-fold form of nested cross-validation, each of the k outer training sets is divided again into l sets for an inner loop.[1] RapidMiner's Cross Validation operator delivers, from its performance ports, the average of the performances over the folds.[4] Treat a cross-validated score as an estimate for the fitting procedure, and refit on all the rows for the model you keep. (Meridian's view)

Compared with the alternatives

Unlike repeated random sub-sampling, k-fold cross-validation uses every observation for both training and validation, and each observation for validation exactly once.[1] Repeated random sub-sampling has the advantage that the proportion of the training and validation split does not depend on the number of iterations.[1]

Sources

  1. [1]Cross-validation (statistics) - Wikipediasecondary source
  2. [2]3.1. Cross-validation: evaluating estimator performance — scikit-learn 1.9.1 documentationdocumentation
  3. [3]Bootstrapping Validation - Rapidminer Documentationdocumentation
  4. [4]Cross Validation - Rapidminer Documentationdocumentation