Assemble a modelling pipeline

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

In scikit-learn, a Pipeline lets the fit and predict methods be called once on the data to fit a whole sequence of estimators.^[1] A scikit-learn Pipeline also allows a grid search over the parameters of all the estimators in it at once.^[1]

In scikit-learn, a Pipeline helps avoid leaking statistics from test data into the trained model in cross-validation, by training every step on the same samples.^[1] In Spark MLlib, a Pipeline and its fitted form help to ensure that training and test data go through identical feature processing steps.^[3] Reach for a pipeline whenever a model depends on preprocessing learnt from the data, so the preprocessing is fitted and applied with the model rather than by hand. (Meridian's view) Treat the order of the steps as part of the model, and choose it deliberately. (Meridian's view)

The method

In scikit-learn, every step of a Pipeline except the last must be a transformer, with a transform method; the last step may be a transformer, a predictor or a clustering estimator.^[1] A scikit-learn Pipeline exposes the methods of its last step: to predict, it transforms the data with every step but the last and gives the result to the last step's predict method.^[1] In scikit-learn, the parameters of a step inside a Pipeline are reached with the step's name, two underscores and the parameter's name.^[1]

Spark MLlib's pipeline concept is mostly inspired by the scikit-learn project.^[3] In Spark MLlib, a Transformer is an algorithm that transforms one DataFrame into another, and an Estimator is an algorithm fit on a DataFrame to produce a Transformer.^[3] When a Spark MLlib Pipeline is fit, its stages run in order: a Transformer stage transforms the DataFrame, and an Estimator stage is fit to produce a Transformer.^[3] A Spark MLlib Pipeline is itself an Estimator, so fitting a Pipeline produces a PipelineModel, which is a Transformer.^[3] In a Spark MLlib PipelineModel, every Estimator of the original Pipeline has become a Transformer, and test data are passed through its stages in order.^[3] A Spark MLlib Pipeline need not be linear: its stages may form a directed acyclic graph, as long as they are specified in topological order.^[3]

In RapidMiner, the model supplied at the first input port of Group Models is the first one applied when the grouped model is applied.^[2] In RapidMiner, a grouped model can be applied like any other model with the Apply Model operator, and written to a file with the Write Model operator.^[2] RapidMiner's Ungroup Models operator splits a grouped model back into its single models, which can then be handled on their own.^[4]

Compared with the alternatives

In scikit-learn, Pipeline is often used with ColumnTransformer or FeatureUnion, which concatenate the outputs of transformers into a composite feature space.^[1]

Sources

1. [1]8.1. Pipelines and composite estimators — scikit-learn 1.9.1 documentationdocumentation
2. [2]Group Models - Rapidminer Documentationdocumentation
3. [3]ML Pipelines - Spark 4.3.0 Documentationdocumentation
4. [4]Ungroup Models - Rapidminer Documentationdocumentation