Model / evaluate · All operations

Evaluate a model

Performance metrics, confusion matrix, ROC, lift, cross-validation.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

Shape of the result
a table describing the input rather than holding its rows
Level
several clauses of SQL, or a window

The DuckDB step

A confusion matrix and accuracy from predictions

SQL
CREATE TEMP TABLE scored AS SELECT * FROM (VALUES ('y','y'),('y','n'),('n','n'),('n','n'),('y','y'),('n','y')) t(actual, predicted);
SELECT actual, predicted, count(*) AS n FROM scored GROUP BY ALL ORDER BY ALL;
CREATE TEMP TABLE result AS
SELECT round(avg((actual = predicted)::INT), 3) AS accuracy,
       round(count(*) FILTER (WHERE actual = 'y' AND predicted = 'y') / count(*) FILTER (WHERE predicted = 'y'), 3) AS precision_y,
       round(count(*) FILTER (WHERE actual = 'y' AND predicted = 'y') / count(*) FILTER (WHERE actual = 'y'), 3) AS recall_y
FROM scored;
SELECT * FROM result;
SELECT CASE WHEN ((SELECT accuracy = 0.667 AND precision_y = 0.667 AND recall_y = 0.667 FROM result)) THEN 'as expected' ELSE error('four right of six did not give 0.667') END AS verdict;
  • The statement builds its own data.
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “four right of six did not give 0.667”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

Variants

Time a model

Measure how long a model takes to train, read data or predict.

Combine performance results

Average, weight or convert several performance results into one result or a table.

Backtest a forecast

Validate a forecasting model over sliding windows or holdout periods of a time series.

Compute performance metrics

Compare actual and predicted values to give accuracy, error measures or a confusion matrix.

Read the evaluation made in training

Return the out-of-bag, validation or per-iteration metrics recorded while the model trained.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Alteryx Designer (5)
  • Cross-Validationread 2026-09-27

    Compare one or more predictive models by k-fold cross-validation

  • K-Centroids Diagnosticsread 2026-09-27

    Assess the right number of clusters for a dataset and clustering algorithm

  • Lift Chartread 2026-09-27

    Draw cumulative gains and incremental response charts to compare binary classifiers

  • Model Comparisonread 2026-09-27

    Compare models on a validation set: error measures, confusion matrix and plots

  • TS Compareread 2026-09-27

    Compare ARIMA and ETS models against holdout data with accuracy measures and plots

KNIME Analytics Platform (12)
  • Enrichment Plotter (legacy)read 2026-09-27

    Draw enrichment curves showing how fast hits accumulate when rows are ranked by a score

  • Entropy Scorerread 2026-09-27

    Score a clustering against a reference clustering by entropy and quality measures

  • KNIME Giskard Extension (extension)read 2026-09-27

    Extension outside the base install: 1 current node: Giskard ML Scanner

  • Lift Chart (JavaScript) (legacy)read 2026-09-27

    Draw a lift chart for a classifier with the legacy JavaScript view

  • Lift Chart (legacy)read 2026-09-27

    Draw a lift chart for a classifier (Swing view)

  • Numeric Scorerread 2026-09-27

    Compute R², MAE, MSE, RMSE and related error statistics between actual and predicted values

  • ROC Curveread 2026-09-27

    Draw ROC curves for classifier scores

  • ROC Curve (JavaScript) (legacy)read 2026-09-27

    Draw ROC curves with the legacy JavaScript view

  • ROC Curve (legacy)read 2026-09-27

    Draw ROC curves for a binary classifier's scores (Swing view)

  • Scorerread 2026-09-27

    Build a confusion matrix and accuracy statistics from actual and predicted class columns

  • X-Aggregatorread 2026-09-27

    Close a cross-validation loop, collecting predictions and error rates per fold

  • X-Partitionerread 2026-09-27

    Open a cross-validation loop, splitting rows into training and test folds per iteration

mlpack (2)
  • Cross-validationread 2026-09-27

    KFoldCV and SimpleCV: score any mlpack method over k folds or one hold-out split

  • Performance measuresread 2026-09-27

    5 measures usable in cross-validation: Accuracy, F1, MSE, Precision, Recall

RapidMiner Studio (33)
Yggdrasil Decision Forests (15)
  • Evaluationread 2026-09-27

    Result of evaluate: loss, accuracy, confusion matrix, RMSE, NDCG, MRR, MAP, Qini, AUUC, custom metrics

  • Go benchmark_inferenceread 2026-09-27

    Go command that times model inference over a dataset

  • benchmarkread 2026-09-27

    Time the model's inference over a dataset, per example and per engine

  • benchmark_inferenceread 2026-09-27

    Time a model's inference over a dataset on every compatible engine

  • benchmark_trainingread 2026-09-27

    Time the training of several learners on fixed datasets and write the results as CSV

  • compare_benchmarkread 2026-09-27

    Python script comparing two training-benchmark result files

  • cross_validationread 2026-09-27

    Train and score the learner over k folds and return one pooled evaluation

  • evaluateread 2026-09-27

    Compute metrics on a labelled dataset: accuracy, confusion matrix, RMSE, NDCG, Qini, bootstrap intervals

  • evaluateread 2026-09-27

    Score a model on a labelled typed-path dataset and print the evaluation

  • evaluate_predictionsread 2026-09-27

    Score any array of predictions against labels, from a YDF model or not

  • out_of_bag_evaluationsread 2026-09-27

    Return the out-of-bag evaluation after each tree; alias of training_logs

  • self_evaluationread 2026-09-27

    Return the evaluation made during training: out-of-bag for forests, validation for boosting

  • training_logsread 2026-09-27

    Return the metrics recorded at each training iteration

  • validation_loss / validation_evaluationread 2026-09-27

    Return the loss and metrics on the validation split held out in training

  • ydf.internal.benchmark_io_speedread 2026-09-27

    Time how fast each dataset format is read into YDF

Background

To evaluate a classifier, its output is compared with a reference classification, ideally a perfect one, and the comparison is summarised in statistics.[1] In a confusion matrix, rows hold the actual classes and columns the predicted ones, or the reverse, so the diagonal holds every correct prediction.[2]

When to use it

Scikit-learn treats learning a model's parameters and testing it on the same data as a methodological mistake, since a model that repeated the labels it had seen would score perfectly.[3] In scikit-learn the common practice is to hold out part of the available data as a test set.[3] Scikit-learn warns that tweaking a model's settings, its hyperparameters, until it performs best on the test set lets knowledge of that set leak into the model.[3] Keep one final test set that no tuning decision has looked at, and score the chosen model on it once. (Meridian's view)

Accuracy yields misleading results when a data set is unbalanced, with very different numbers of observations in its classes.[2] With 95 cancer samples and 5 non-cancer samples, a classifier that labels every observation as cancer has an overall accuracy of 95% but a 0% recognition rate for the non-cancer class.[2] An important distinction among classifier metrics is between those independent of prevalence, how often each class occurs in the population, and those that depend on it.[1] Different fields prefer different metrics: medicine often uses sensitivity and specificity, while computer science prefers precision and recall.[1] Scikit-learn advises that where a scoring function is given, for example by a business context, that is the one to use.[4] Choose the metric from the decision the model will feed, and show the confusion matrix beside any single figure so a reader can see where the errors fall. (Meridian's view)

The method

The four outcomes of a binary classifier are true positives, false positives, true negatives and false negatives: a false positive is predicted positive when the actual value is negative.[5]

For numeric predictions, KNIME's Numeric Scorer computes R² as one minus the residual sum of squares over the total sum of squares, and notes that it can be negative.[6] KNIME's Numeric Scorer also reports the mean absolute error, the mean squared error and the root mean squared error between actual and predicted values.[6]

Compared with the alternatives

When evaluation is used to choose between two classification methods, a single unitary metric makes the comparison more directly than a pair of metrics.[1] Compare models on one metric, computed the same way on the same held-out rows; figures from different metrics or different rows do not rank models. (Meridian's view)

Sources

  1. [1]Evaluation of binary classifiers - Wikipediasecondary source
  2. [2]Confusion matrix - Wikipediasecondary source
  3. [3]3.1. Cross-validation: evaluating estimator performance — scikit-learn 1.9.1 documentationdocumentation
  4. [4]3.4. Metrics and scoring: quantifying the quality of predictions — scikit-learn 1.9.1 documentationdocumentation
  5. [5]Receiver operating characteristic - Wikipediasecondary source
  6. [6]Numeric Scorerdocumentation