Search for the best set of columns

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

Testing every possible subset of features is an exhaustive search, computationally intractable for all but the smallest feature sets.^[1] Stepwise regression is a greedy algorithm that adds the best feature, or deletes the worst, at each round.^[1] Reach for a stepwise or greedy search when there are too many columns to try every combination, and when the score of the model you will actually use is the measure you trust. (Meridian's view)

Wrappers can be computationally expensive and carry a risk of overfitting to the model.^[1] Stepwise regression searches a large space of possible models, so it is prone to overfitting, often fitting much better in sample than on new out-of-sample data.^[2] Reporting estimates and confidence intervals from the final stepwise model without adjusting them for the model-building process has led to calls to stop using stepwise model building, or at least to reflect model uncertainty correctly.^[2] Treat the chosen set of columns as one outcome of a search, judge it on data the search never saw, and do not read its p-values as if the columns had been chosen in advance. (Meridian's view)

Many common criteria for choosing a feature subset combine a measure of accuracy with a penalty for the number of features, such as the Akaike information criterion (AIC), which has a penalty of 2 for each added feature.^[1] The Stepwise tool in Alteryx Designer offers the AIC or the Bayesian information criterion (BIC) as its fit measure, and the BIC typically gives a final model with fewer variables because it penalises each variable more.^[3]

The method

Forward selection starts with no variables, tests adding each one using a chosen model fit criterion, adds the one that most improves the fit, and repeats until no addition improves the model to a statistically significant extent.^[2] Backward elimination starts with all candidate variables, deletes the one whose loss gives the most statistically insignificant deterioration of the fit, and repeats until no variable can be deleted without a statistically significant loss of fit.^[2] Bidirectional elimination combines forward selection and backward elimination, testing at each step for variables to be included or excluded.^[2] A widely used stepwise algorithm, a variation on forward selection that checks after each addition whether some variables can be deleted, was first proposed by Efroymson (1960).^[2]

In scikit-learn's SequentialFeatureSelector, forward and backward selection do not in general give the same result, and one may be much faster than the other depending on how many features are requested.^[4] Besides greedy forward selection and greedy backward elimination, search approaches for feature selection include exhaustive search, simulated annealing, genetic algorithms and particle swarm optimisation.^[1] In KNIME's feature selection loop, columns can be held static, included in every iteration and exempt from elimination, while a forward, backward, genetic or random strategy works on the rest.^[5] In Yggdrasil Decision Forests (YDF), the BackwardSelectionFeatureSelector removes a ratio of the features at each iteration, 0.1 by default, guided by a variable importance, and returns the best model it trained.^[6]

Compared with the alternatives

A filter scores features with a fast proxy measure instead of a model's error rate, so its feature set is not tuned to a specific type of predictive model and usually gives lower prediction performance than a wrapper's.^[1] A filter's feature set does not contain the assumptions of a prediction model, which makes it more useful than a wrapper's for exposing the relationships between the features.^[1] Filter methods have also been used as a preprocessing step for wrapper methods, allowing a wrapper to be used on larger problems.^[1] In scikit-learn, sequential feature selection differs from recursive feature elimination (RFE) and SelectFromModel in not requiring the model to expose a coef_ or feature_importances_ attribute, though it may be slower because more models must be evaluated.^[4]

Sources

1. [1]Feature selection - Wikipediasecondary source
2. [2]Stepwise regression - Wikipediasecondary source
3. [3]Stepwise Tooldocumentation
4. [4]1.13. Feature selection — scikit-learn 1.9.1 documentationdocumentation
5. [5]Feature Selection Loop Start (1:1)documentation
6. [6]yggdrasil-decision-forests/yggdrasil_decision_forests/port/python/ydf/learner/feature_selector.py at 0a54b57c95ff0cf42ecf0310ace0513a4f32cb65 · google/yggdrasil-decision-forests · GitHubdocumentation