Train a tree model

Status

Variants

Train a single decision tree

Learn one tree of if-then splits that predicts a class or a number.

Choose a classification or regression task

Set whether the tree predicts a category or a number.

Train from code or the command line

Call the training step from an API, a command-line tool or a browser package rather than a visual operator.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

Decision trees are simple to understand and interpret, and can be displayed graphically in a way that non-experts can read.^[1][2] Decision trees need little data preparation: other techniques often require data normalisation, and a tree can handle qualitative predictors without dummy variables.^[1] In Alteryx's Forest Model and Boosted Model tools, columns holding unique identifiers such as primary keys should not be used as predictors, since they have no predictive value.^[3][4] Whatever the tool, leave row identifiers out of a tree's predictors. (Meridian's view)

Decision-tree learners can create over-complex trees that do not generalise well from the training data, which is known as overfitting.^[1][2] In scikit-learn, the max_depth parameter controls the size of a decision tree to prevent overfitting.^[2] In scikit-learn, decision trees tend to overfit on data with a large number of features, and getting the right ratio of samples to number of features is important.^[2] Trees can be very non-robust: a small change in the training data can result in a large change in the tree and in its final predictions.^[1] In scikit-learn, decision tree predictions are piecewise constant approximations, neither smooth nor continuous, so the trees are not good at extrapolation.^[2] Do not ask a tree for predictions far outside the range of the rows it was trained on. (Meridian's view) In scikit-learn, decision tree learners create biased trees if some classes dominate.^[2] Balance or weight the classes before fitting a tree to data where one class dominates. (Meridian's view)

Reach for a single tree when the people using a model need to read and question its rules, and for an ensemble of trees when accuracy matters more than reading them. (Meridian's view)

The method

A tree is built by splitting the source set, which forms the root node, into subsets that become its children, using splitting rules based on the features.^[1] The splitting is repeated on each derived subset, which is called recursive partitioning, until a node's subset has one value of the target or splitting no longer adds value to the predictions.^[1] Top-down induction of decision trees (TDIDT) is an example of a greedy algorithm, and by far the most common strategy for learning decision trees from data.^[1]

Algorithms for constructing decision trees usually choose, at each step, the variable that best splits the set of items, and different algorithms use different metrics for measuring the best split.^[1] The term classification and regression tree (CART) analysis covers both classification and regression trees, and was first introduced by Breiman et al. (1984).^[1] Gini impurity, used by the CART algorithm for classification trees, is the probability that a randomly chosen element of a set would be mislabelled if it were labelled randomly according to the set's distribution of labels.^[1] Information gain, used by the ID3, C4.5 and C5.0 tree-generation algorithms, is based on the concept of entropy from information theory.^[1] For data including categorical variables with different numbers of levels, information gain is biased in favour of attributes with more levels.^[1] One counter to the bias of information gain is to choose the attribute with the highest information gain ratio among the attributes whose information gain is greater than the mean information gain.^[1]

Compared with the alternatives

A decision tree is a white box model, whose explanation for a result is easily given by Boolean logic, unlike a black box model such as an artificial neural network.^[1][2] Boosted trees build an ensemble incrementally, training each new tree to emphasise the training instances previously mis-modelled.^[1] Bootstrap aggregated, or bagged, decision trees are built by repeatedly resampling the training data with replacement, and voting the trees for a consensus prediction.^[1] While random forests often achieve higher accuracy than a single decision tree, they sacrifice the intrinsic interpretability of decision trees.^[5] Boosting can increase the accuracy of a base learner such as a decision tree, but sacrifices intelligibility and interpretability.^[6] In RapidMiner, the Decision Tree operator builds one tree, which makes the prediction more comprehensible for humans but might lead to overtraining.^[7]

Sources

1. [1]Decision tree learning - Wikipediasecondary source
2. [2]1.10. Decision Trees — scikit-learn 1.9.1 documentationdocumentation
3. [3]Forest Model Tooldocumentation
4. [4]Boosted Model Tooldocumentation
5. [5]Random forest - Wikipediasecondary source
6. [6]Gradient boosting - Wikipediasecondary source
7. [7]Random Forest - Rapidminer Documentationdocumentation