Fit a logistic or generalised linear regression

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

A linear-response model is unsuitable for predicting the probability of a yes-or-no choice, because probabilities are bounded and must lie between 0 and 1.^[1] For a binary response, the most typical link function is the logit link, and GLMs with that setup are logistic regression models.^[1] In logistic regression the coefficients represent the change in the logit for each unit change in a predictor, whereas in linear regression they represent the change in the outcome.^[2] The defining characteristic of the logistic model is that increasing one of the independent variables multiplicatively scales the odds of the outcome at a constant rate.^[2]

A predicted number of beach attendees, a count, would typically be modelled with a Poisson distribution and a log link.^[1] Poisson regression may also suit rate data, where a count of events is divided by a measure of each unit's exposure, such as area, person-years or time.^[3] In Poisson regression, exposure is handled as an offset, log(exposure), which enters the model as a term whose parameter estimate is constrained to 1.^[3] The Poisson distribution's mean is equal to its variance, and an observed variance greater than the mean, known as overdispersion, indicates that the Poisson model is not appropriate.^[3] Under some circumstances, overdispersion can be handled with quasi-likelihood estimation or a negative binomial distribution.^[3] Compare the variance of a count target with its mean before trusting a Poisson fit. (Meridian's view)

In Alteryx, gamma regression fits a target that is always strictly positive and tends to cluster toward the lower range of its values, estimated through the generalised linear model framework.^[4] Reach for a gamma model when an amount can never be zero or negative and a few large values sit far above the rest. (Meridian's view)

The coefficients of a logistic regression have no closed-form expression that maximises the likelihood, so an iterative process such as Newton's method must be used instead.^[2] Multicollinearity refers to unacceptably high correlations between predictors.^[2] A logistic regression may fail to converge, for reasons including a large ratio of predictors to cases, multicollinearity, sparseness or complete separation.^[2] Non-convergence of a logistic regression indicates that its coefficients are not meaningful, because the iterative process could not find appropriate solutions.^[2] Complete separation is where the predictors perfectly predict the outcome, and the likelihood is maximised with infinite coefficients.^[2] Check that a fit reports convergence before reading its coefficients, and when it does not, look for a predictor that splits the outcome perfectly. (Meridian's view)

The method

A GLM consists of three elements: a distribution for the outcome from an exponential family, a linear predictor, and a link function.^[1] The exponential family is a large class of probability distributions that includes the normal, binomial, Poisson and gamma distributions, among others.^[1] The link function of a GLM provides the relationship between the linear predictor and the mean of the distribution, and must be bijective so that its inverse exists.^[1] A GLM always has a well-defined canonical link function, derived from the exponential of the response's density function.^[1] For the gamma distribution, the domain of the canonical link function is not the same as the permitted range of the mean, so the linear predictor may give an impossible negative mean.^[1] One alternative to the canonical link for a gamma model is a non-canonical link function.^[1] For a positive, skewed amount, consider a log link rather than the canonical one. (Meridian's view)

John Nelder and Robert Wedderburn proposed an iteratively reweighted least squares (IRLS) method for maximum likelihood estimation (MLE) of a GLM's parameters.^[1] MLE remains popular and is the default method on many statistical computing packages.^[1] For the normal distribution, a GLM has a closed-form expression for the maximum-likelihood estimates, but most other GLMs lack closed-form estimates.^[1] Results for a GLM with a non-identity link are asymptotic, tending to work well with large samples.^[1] The stats_duck DuckDB community extension fits a GLM as an aggregate with glm_fit, for the gaussian, binomial, poisson and gamma families.^[5]

For a binary response, the inverse of any continuous cumulative distribution function can also be used as the link, and the normal one yields the probit model.^[1] The logistic model was likely first used as an alternative to the probit model in bioassay, by Edwin Bidwell Wilson and Jane Worcester in 1943.^[2] Joseph Berkson coined the term logit in 1944, by analogy with probit, and over many decades developed the logistic model as a general alternative to the probit model.^[2]

Compared with the alternatives

The general linear model may be viewed as a special case of the GLM with an identity link and normally distributed responses.^[1] John Nelder, a co-originator of GLMs, has expressed regret over the terminology that makes generalised linear models easy to confuse with general linear models.^[1]

Negative binomial regression is a popular generalisation of Poisson regression that loosens the assumption that the variance is equal to the mean.^[3] If the mean is μ, the quasi-Poisson model assumes a variance of θμ while the negative binomial, or gamma-Poisson, model assumes μ(1 + κμ), where θ is an overdispersion parameter and κ a shape parameter.^[3]

Sources

1. [1]Generalized linear model - Wikipediasecondary source
2. [2]Logistic regression - Wikipediasecondary source
3. [3]Poisson regression - Wikipediasecondary source
4. [4]Gamma Regression Tooldocumentation
5. [5]stats_duck – DuckDB Community Extensionsdocumentation