Calibrate a model's confidence

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

Some models that do give probabilities, notably naive Bayes classifiers, decision trees and boosting methods, produce distorted class probability distributions.^[1] In scikit-learn's calibration example, Gaussian naive Bayes tends to push probabilities to 0 or 1, while a random forest rarely gives probabilities close to 0 or 1.^[3] Max-margin methods such as support vector machines (SVMs) and boosted trees show sigmoidal distortions in their predicted probabilities.^[2]

A calibration plot, also called a calibration curve or reliability diagram, shows the proportion of each class for bands of predicted probability or score.^[1][3] On a calibration plot, deviations from the identity function indicate a poorly calibrated classifier whose scores cannot be used as probabilities.^[1] Draw the calibration plot before and after calibrating, rather than trusting a single score to show the change. (Meridian's view)

In RapidMiner, rescaling confidences is worth considering when optimising a measure other than area under the curve (AUC), with cost-sensitive learning, or with highly imbalanced classes.^[4] Even after rescaling in RapidMiner, there is no guarantee that the confidences are perfect probabilities.^[4]

The method

In scikit-learn, calibrating a classifier fits a regressor, called a calibrator, that maps the classifier's output to a calibrated probability between 0 and 1.^[3] Platt scaling gives the probability as 1 / (1 + exp(A f(x) + B)) for a classifier output f(x), with the two scalar parameters A and B learned by maximum likelihood.^[2][3] To avoid overfitting, Platt scaling can be fitted on a held-out calibration set or by cross-validation, and Platt also suggested replacing the labels with smoothed target probabilities.^[2] Platt suggested the Levenberg–Marquardt algorithm to fit the parameters, and a Newton algorithm, meant to be more numerically stable, was proposed later.^[2]

In scikit-learn, the sigmoid method is based on Platt's logistic model and assumes a sigmoid function applied to the raw predictions can correct the calibration curve.^[3] An alternative to Platt scaling fits an isotonic regression model to an ill-calibrated probability model.^[2][1] In scikit-learn, isotonic calibration fits a step-wise non-decreasing function, so it can correct any monotonic distortion of the uncalibrated model.^[3] In scikit-learn, isotonic calibration is more prone to overfitting than the sigmoid method, especially on small datasets.^[3] In scikit-learn, isotonic calibration is expected to perform as well as or better than the sigmoid method when there are more than about 1000 samples.^[3] In scikit-learn, the sigmoid method is most effective for small sample sizes, or when the uncalibrated model is under-confident with similar errors for high and low outputs.^[3]

In scikit-learn, the sigmoid method and isotonic calibration handle several classes by calibrating each class separately against the rest and then normalising the probabilities.^[3] In RapidMiner, Rescale Confidences (Logistic) fits a full logistic regression model for each class, because the simplified original Platt approach worked only for binary problems.^[4]

Compared with the alternatives

In scikit-learn, calibration is generally not expected to change ranking metrics such as AUC, though isotonic calibration might, because it introduces ties in the predicted probabilities.^[3] In scikit-learn, the sigmoid method is a strictly monotonic transformation, so it keeps the ranking and the AUC of the uncalibrated model.^[3] In scikit-learn, tuning the decision threshold leaves a classifier's probabilities unchanged and changes only the class labels it predicts from them.^[5] Calibrate when the probability itself will be read or acted on; tune the threshold when only the yes-or-no decision matters. (Meridian's view)

Sources

1. [1]Probabilistic classificationsecondary source
2. [2]Platt scalingsecondary source
3. [3]1.16. Probability calibration — scikit-learn 1.9.1 documentationdocumentation
4. [4]Rescale Confidences (Logistic) - Rapidminer Documentationdocumentation
5. [5]3.3. Tuning the decision threshold for class prediction — scikit-learn 1.9.1 documentationdocumentation