Train a naive Bayes classifier

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

Naive Bayes classifiers have worked quite well in many real-world situations, famously document classification and spam filtering, and need only a small amount of training data to estimate their parameters.^[2][1] Naive Bayes classifiers are highly scalable, requiring only one parameter for each feature or predictor in a learning problem.^[1] In scikit-learn, ComplementNB implements complement naive Bayes (CNB), an adaptation of multinomial naive Bayes that is particularly suited to imbalanced data sets.^[2]

Naive Bayes classifiers generally perform worse than more advanced models such as logistic regression at quantifying uncertainty, often producing wildly overconfident probabilities.^[1] In scikit-learn, naive Bayes is known as a decent classifier but a bad estimator, so the probability outputs from its predict_proba method are not to be taken too seriously.^[2] A naive Bayes classifier still picks the correct class under the maximum a posteriori (MAP) decision rule as long as the correct class is predicted as more probable than any other, however inaccurate the probability estimates are.^[1] Trust a naive Bayes model's choice of class more than the probability it attaches to that choice, and calibrate the scores before reading them as probabilities. (Meridian's view)

If a class and a feature value never occur together in the training data, the frequency-based probability estimate is zero, which wipes out the information in the other probabilities when they are multiplied.^[1] A small-sample correction called a pseudocount, added to every probability estimate, keeps any probability from being exactly zero.^[1] Regularising naive Bayes with a pseudocount is called Laplace smoothing when the pseudocount is one, and Lidstone smoothing in the general case.^[1] Set a smoothing pseudocount whenever new rows may carry a value the training rows never paired with a class. (Meridian's view)

The method

Naive Bayes applies Bayes' theorem, under which a class's posterior probability given the features is proportional to its prior probability times the likelihood of the features given the class.^[1][2] Under the naive independence assumption, the likelihood becomes a product of one probability per feature, and the classifier picks the class with the largest product of prior and per-feature probabilities.^[1][2] A class's prior may be calculated by assuming equiprobable classes, or estimated as the share of training samples in that class.^[1] Working with log-probabilities, a sum of the log prior and the log likelihoods, avoids arithmetic underflow.^[1] Because each class-conditional feature distribution can be estimated independently as a one-dimensional distribution, naive Bayes alleviates problems stemming from the curse of dimensionality.^[1][2]

The assumptions about the distributions of features are called the event model of the naive Bayes classifier.^[1] Gaussian naive Bayes, for continuous data, segments the training data by class and computes the mean and variance of each feature in each class, then scores a value with the normal density of those parameters.^[1] Where the class-conditional marginal densities are far from normal, kernel density estimation, introduced by John and Langley, can give a more realistic estimate of the densities of each class.^[1] With a multinomial event model, a feature vector is a histogram counting how many times each event was observed, the event model typically used for document classification with words as events.^[1] In the multivariate Bernoulli event model, features are independent Boolean variables recording whether each term occurs, which has the benefit of explicitly modelling the absence of terms.^[1]

Bayesian algorithms were used for email filtering as early as 1996, and the first scholarly publication using the naive Bayes classifier for spam filtering was by Sahami et al. in 1998.^[1]

Compared with the alternatives

On discrete inputs, naive Bayes and multinomial logistic regression form a generative-discriminative pair: naive Bayes fits a probability model optimising the joint likelihood of class and features, logistic regression the same model optimising the conditional likelihood of the class.^[1] Discriminative classifiers have lower asymptotic error than generative ones, but research by Ng and Jordan has shown that in some practical cases naive Bayes can outperform logistic regression because it reaches its asymptotic error faster.^[1] The multinomial naive Bayes classifier becomes a linear classifier when expressed in log-space.^[1] A comprehensive comparison with other classification algorithms in 2006 showed that naive Bayes classification is outperformed by other approaches, such as boosted trees or random forests.^[1]

Sources

1. [1]Naive Bayes classifiersecondary source
2. [2]1.9. Naive Bayesdocumentation