Train a linear model

Status

Variants

Train a discriminant analysis classifier

Train a linear, quadratic or regularised discriminant analysis classifier.

Train a linear regression to predict a number

Train a linear regression, plain or regularised, to predict a number.

Fit a logistic regression

Train a logistic regression classifier for two or more classes.

Fit another regression-family model

Train splines, polynomial, Gaussian process and other models outside the main families.

Train a perceptron

Train a single-layer perceptron that finds a separating hyperplane.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

Linear classifiers work well for practical problems such as document classification, and more generally for problems with many features, reaching accuracy comparable to non-linear classifiers while taking less time to train and use.^[1] A linear classifier is often used where the speed of classification is an issue, since it is often the fastest classifier, especially when the feature vector is sparse.^[1] When the number of features is large, as in document classification, a linear classifier should be well regularised.^[1]

For scikit-learn's ordinary least squares (OLS), correlated features make the estimate highly sensitive to random errors in the observed target, a situation called multicollinearity.^[3] Ridge regression and other penalised estimation, such as lasso regression, deliberately introduce bias into the estimates to reduce their variability.^[2] Penalised estimates generally have lower mean squared error than OLS estimates, particularly when multicollinearity is present or when overfitting is a problem.^[2] Penalised methods are generally used to predict the response for predictor values not yet observed, and are not as commonly used for inference, since the bias is difficult to account for.^[2] Decide first whether the model is for predicting new rows or for explaining the rows you have, because that choice decides whether a penalty belongs in the fit. (Meridian's view)

In scikit-learn, logistic regression applies regularisation by default, which is common in machine learning but not in statistics.^[3] Before reading a fitted model's coefficients as plain estimates, check whether the tool penalised them by default. (Meridian's view)

The method

A linear classifier's output score is a function of the dot product of a weight vector and the feature vector, and often that function is a threshold that maps scores above it to one class and the rest to the other.^[1] For a two-class problem, a linear classifier can be pictured as splitting the input space with a hyperplane, classifying the points on one side as yes and the others as no.^[1] There are two broad classes of methods for determining a linear classifier's weights: generative models, which model the joint probability distribution, and discriminative models, which model the conditional density of the class given the features.^[1] Linear discriminant analysis (LDA) and naive Bayes with multinomial or multivariate Bernoulli event models are generative examples, while logistic regression, the perceptron and the support vector machine (SVM) are trained discriminatively.^[1] Discriminative training of a linear classifier usually minimises a loss function measuring the discrepancy between the classifier's outputs and the desired outputs, plus a regularisation function that keeps the parameters from getting too large.^[1] LDA, naive Bayes, logistic regression, the perceptron and the SVM can each be converted into a non-linear algorithm operating on a different input space, using the kernel trick.^[1]

Least squares linear regression, as a means of finding a good rough linear fit to a set of points, was performed by Legendre (1805) and Gauss (1809) for the prediction of planetary movement.^[2] Under least squares, the coefficients of a linear regression model are estimated by minimising a sum of squared errors.^[4] Because the sum of squares to be minimised is a quadratic function of the coefficients, minimising it is a relatively simple problem.^[4] In a linear regression model, the word linear refers to the coefficients appearing in a linear way, so the independent variables may enter through nonlinear functions.^[4] In scikit-learn, logistic regression is a special case of generalised linear models with a binomial or Bernoulli conditional distribution and a logit link, and its predicted probability becomes a class by a threshold of 0.5 by default.^[3]

Compared with the alternatives

In scikit-learn's terms, logistic regression is a linear model for classification rather than regression, despite its name.^[3] Logistic regression is also known in the literature as logit regression, maximum-entropy classification (MaxEnt) or the log-linear classifier.^[3] In scikit-learn, ridge regression adds a penalty on the size of the coefficients to the least-squares fit, and a larger value of its complexity parameter, alpha, shrinks the coefficients more and makes them more robust to collinearity.^[3]

Sources

1. [1]Linear classifiersecondary source
2. [2]Linear regressionsecondary source
3. [3]1.1. Linear Modelsdocumentation
4. [4]Linear modelsecondary source