Train a support vector machine

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

SVMs are helpful in text and hypertext categorisation, and classification of images and recognition of hand-written characters can also be performed using SVMs.^[1] In scikit-learn, SVMs are effective in high-dimensional spaces, including cases where the number of dimensions is greater than the number of samples.^[2]

In scikit-learn, SVM algorithms are not scale invariant, so scaling each attribute, for example to [0,1] or [-1,+1] or to mean 0 and variance 1, is highly recommended, with the same scaling applied to the test vectors.^[2] Fit the scaling on the training rows and store it beside the model, so that new rows are scaled the same way before they are scored. (Meridian's view) Drawbacks of the SVM include uncalibrated class membership probabilities, a solved model whose parameters are difficult to interpret, and direct applicability only to two-class tasks.^[1] In scikit-learn, SVMs do not directly provide probability estimates, which are calculated by Platt scaling, a logistic regression on the SVM's scores fitted by an expensive additional cross-validation.^[2] In scikit-learn, the Platt-scaled probabilities may be inconsistent with the SVM's scores, so a sample may be labelled positive even when its predicted probability is below 0.5.^[2] Where only a ranking or a confidence is needed, use the SVM's scores and leave the probabilities off. (Meridian's view)

The method

For data that are not linearly separable, the soft-margin SVM uses the hinge loss, which is zero for a point on the correct side of the margin and proportional to the distance from the margin for a point on the wrong side.^[1] In the soft-margin SVM, a parameter C determines the trade-off between increasing the margin size and keeping the training points on the correct side of the margin.^[1] In scikit-learn, the penalty term C acts as an inverse regularisation parameter, and decreasing C, which is 1 by default, corresponds to more regularisation, advised when there are many noisy observations.^[2] SVMs belong to a family of generalised linear classifiers, can be interpreted as an extension of the perceptron, and simultaneously minimise the empirical classification error and maximise the geometric margin.^[1]

In 1992, Bernhard Boser, Isabelle Guyon and Vladimir Vapnik suggested a way to create nonlinear classifiers by applying the kernel trick to maximum-margin hyperplanes.^[1] The soft-margin SVM, as commonly used in software packages, was proposed by Corinna Cortes and Vapnik in 1993 and published in 1995.^[1] With the kernel trick, the SVM fits the maximum-margin hyperplane in a transformed feature space, so the classifier, a hyperplane there, may be nonlinear in the original input space.^[1]

Platt's sequential minimal optimization (SMO) algorithm trains an SVM by breaking the problem into two-dimensional sub-problems that are solved analytically.^[1] The special case of the linear SVM can be solved more efficiently by the same kind of algorithms used to optimise logistic regression, such as sub-gradient descent and coordinate descent.^[1]

Compared with the alternatives

Whether SVMs have better predictive performance than other linear models, such as logistic regression and linear regression, is not clear.^[1] Among discriminative linear classifiers, logistic regression estimates the weights by maximum likelihood, the perceptron attempts to fix every error in the training set, and the SVM maximises the margin between the decision hyperplane and the training examples.^[3]

Sources

1. [1]Support vector machinesecondary source
2. [2]1.4. Support Vector Machinesdocumentation
3. [3]Linear classifiersecondary source