Model / supervised · All operations

Combine models

Bagging, boosting, stacking, voting, a choice of learners, AutoML.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

Shape of the result
a model or a file, not a table
Level
needs an arcform operator

Variants

  • Boost a weak learner

    Wrap any learner and train it round after round, reweighting rows or fitting the errors of earlier rounds, and add the rounds into one model; gradient boosted trees fix the learner as a tree.

Choose a learner automatically

Pick from a choice of algorithms, or let an automated search build and compare candidate models.

Combine several learners into one model

Train several models, by bagging or in a loop, and join them by vote, averaging or a meta learner.

Other learning tasks and learner libraries

Ranking and uplift tasks, baseline and updatable models, and add-on libraries of learners.

Wrap a learner to fit a different task

Train one inner model per class, label or group, or on a transformed label or cost matrix, and combine them.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Alteryx Designer (5)
  • Assisted Modelingread 2026-09-27

    Build a machine-learning pipeline step by step or automatically and compare candidates

  • Classificationread 2026-09-27

    Train a classifier: logistic regression, decision tree, random forest or XGBoost

  • DataRobot Automodelread 2026-09-27

    Upload data to DataRobot, which builds and stores predictive models automatically

  • Fitread 2026-09-27

    Fit the assembled machine-learning pipeline to data and output the model

  • Regressionread 2026-09-27

    Train a regressor from a choice of algorithms, with tunable parameters

KNIME Analytics Platform (13)
mlpack (2)
  • AdaBoostread 2026-09-27

    Adaptive boosting classifier over weak learners

  • mlpack_adaboost_trainread 2026-09-27

    Train AdaBoost from a features table and a labels table; write the JSON model to a table; return fitted classes

RapidMiner Studio (16)
  • AdaBoostread 2026-09-27

    Boost an inner learner with AdaBoost

  • Additive Regressionread 2026-09-27

    Boost an inner regression learner by fitting successive models to residuals

  • Baggingread 2026-09-27

    Train an inner learner on bootstrap samples and aggregate the models

  • Bayesian Boostingread 2026-09-27

    Boost an inner learner by reweighting rows according to Bayes' theorem

  • Classification by Regressionread 2026-09-27

    Build a multi-class classifier from one inner regression model per class

  • Default Modelread 2026-09-27

    Build a baseline model that always predicts the mode, mean, median or a constant

  • Hierarchical Classificationread 2026-09-27

    Train classifiers arranged along a user-given class hierarchy

  • MetaCostread 2026-09-27

    Make an inner classifier cost-sensitive using a cost matrix

  • Multi Label Modelingread 2026-09-27

    Train one inner model per label column and combine them into a multi-label model

  • Polynomial by Binomial Classificationread 2026-09-27

    Build a multi-class classifier by combining inner two-class models trained per class

  • Relative Regressionread 2026-09-27

    Train an inner regression on the label expressed relative to another column

  • Stackingread 2026-09-27

    Train base learners and a meta learner on their predictions (stacking)

  • Subset Modelingread 2026-09-27

    Train one inner model per group of rows defined by a column

  • Transformed Regressionread 2026-09-27

    Train an inner regression on a transformed label (log, exp, z-scale or rank)

  • Update Modelread 2026-09-27

    Update an already trained, updatable model with additional rows

  • Voteread 2026-09-27

    Combine inner learners by majority vote or averaged prediction

Yggdrasil Decision Forests (4)
  • CATEGORICAL_UPLIFTread 2026-09-27

    Predict the incremental effect of a treatment on a categorical outcome

  • MULTITASKERread 2026-09-27

    Meta-learner that trains several models in parallel and returns one model holding them all

  • NUMERICAL_UPLIFTread 2026-09-27

    Predict the incremental effect of a treatment on a numerical outcome

  • RANKINGread 2026-09-27

    Order items within a group by a relevance label from 0 to 4, scored by NDCG

Background

An ensemble method uses several learning algorithms together to get better predictive performance than any one of them could give alone.[1] Ensemble learning typically refers to bagging (bootstrap aggregating), boosting, or stacking.[1] Evaluating an ensemble's prediction typically takes more computation than evaluating the prediction of a single model.[1]

When to use it

Bagging was designed to improve the stability and accuracy of classification and regression algorithms, and it also reduces variance and overfitting.[2] Bagging leads to improvements for unstable procedures, such as artificial neural networks and classification and regression trees, but can mildly degrade stable methods such as k-nearest neighbours.[2] Ensembles tend to yield better results when there is significant diversity among the models they combine.[1] Using a variety of strong learning algorithms has been shown to be more effective than techniques that dumb down the models to promote diversity.[1]

In scikit-learn, a VotingClassifier combines conceptually different classifiers by majority vote or by averaging their predicted probabilities, and can be useful for a set of equally well performing models, to balance out their individual weaknesses.[3] In scikit-learn, a stacking predictor predicts as well as the best predictor of its base layer and sometimes better, but training it is computationally expensive.[3] A bucket of models uses a model selection algorithm to choose the best model for each problem, and tested on only one problem it can produce no better results than the best model in the set.[1]

Reach for an ensemble when a single model's predictions swing with small changes in the training rows, and keep a single model when someone has to follow how each prediction was made. (Meridian's view) Judge an ensemble on rows held out from every member's training, exactly as you would judge a single model. (Meridian's view)

The method

Given a training set of size n, bagging draws m new training sets by sampling uniformly and with replacement, fits a model to each, and combines them by averaging for regression or voting for classification.[2] Bootstrap aggregating was proposed by Leo Breiman, who coined the term bagging and developed the concept in 1994.[2] Random forests are a common application of bagging.[1]

Boosting trains successive models by emphasising the training data that previously learned models misclassified.[1]

Stacking, sometimes called stacked generalization, trains a combiner algorithm to make a final prediction from the predictions of several other learning algorithms.[1] In stacking, the combiner can be trained on cross-validated predictions from the base estimators, which can prevent overfitting, and in practice a logistic regression model is often used as the combiner.[1] In scikit-learn's StackingClassifier and StackingRegressor, the final estimator is trained on out-of-sample predictions made internally with cross_val_predict, to generalise and avoid over-fitting.[3]

In scikit-learn's VotingClassifier with hard voting, the predicted class is the one most of the classifiers predict, and a tie goes to the class first in ascending sort order.[3] Bayesian model averaging (BMA) makes predictions by averaging the predictions of models weighted by their posterior probabilities given the data.[1]

Compared with the alternatives

Bagging fits the same model to different random samples of the training data, a homogeneous parallel ensemble, while stacking combines different base models trained independently into a heterogeneous parallel ensemble.[1] Cross-validation selection, sometimes called a bake-off contest, tries every model on the training set and keeps the one that works best.[1] RapidMiner's Stacking operator uses performance on held-out data to combine its base models rather than choose among them.[4] BMA is known generally to give better answers than a single model obtained, for example, by stepwise regression, especially where very different models perform nearly identically on the training set.[1]

Sources

  1. [1]Ensemble learning - Wikipediasecondary source
  2. [2]Bootstrap aggregating - Wikipediasecondary source
  3. [3]1.11. Ensembles: Gradient boosting, random forests, bagging, voting, stacking — scikit-learn 1.9.1 documentationdocumentation
  4. [4]Stacking - Rapidminer Documentationdocumentation