Search hyperparameters

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

Grid search suffers from the curse of dimensionality, but is often embarrassingly parallel because the hyperparameter settings it evaluates are typically independent of each other.^[1] Grid search may need manually set bounds and discretisation for hyperparameters whose values are real-valued or unbounded.^[1] Keep a grid to the two or three settings that matter and a handful of values for each; a grid that grows by multiplication is easy to start and slow to finish. (Meridian's view)

Random search can outperform grid search, especially when only a small number of hyperparameters affects the final performance of the algorithm.^[1] James Bergstra and Yoshua Bengio found that for most data sets only a few hyperparameters really matter, but that different hyperparameters matter on different data sets, which makes grid search a poor choice for new data sets.^[2] By default, a scikit-learn parameter search scores each setting with the estimator's own score function, which for classification is accuracy, and accuracy is often uninformative for unbalanced classification.^[3]

The method

Cross-validation is often used to estimate the generalisation performance of each candidate set, so that the set of values that maximises it can be chosen.^[1] Random search replaces the exhaustive enumeration of all combinations by selecting them randomly, and generalises to continuous and mixed spaces.^[1]

Bayesian optimisation builds a probabilistic model of the function from hyperparameter values to the objective on a validation set, and uses it to choose each next configuration, balancing exploration against exploitation.^[1] In practice, Bayesian optimisation has been shown to obtain better results in fewer evaluations than grid search and random search.^[1] Evolutionary optimisation ranks a population of hyperparameter sets by fitness and replaces the worst with new sets made by crossover and mutation, repeating until performance is satisfactory or stops improving.^[1] Successive halving begins as a random search but periodically prunes low-performing models, focusing computation on the more promising ones.^[1] In scikit-learn's successive halving, all candidates are first evaluated with a small amount of resources, typically the number of training samples, and only some go on to the next iteration with more.^[3] For specific learning algorithms, the gradient with respect to the hyperparameters can be computed and the hyperparameters optimised by gradient descent.^[1]

mlpack's HyperParameterTuner uses grid search by default, and can use a gradient-based optimiser instead to tune a hyperparameter over all real values.^[4]

Compared with the alternatives

Population Based Training (PBT) is adaptive, updating hyperparameters while the models train, whereas non-adaptive methods assign one constant set of hyperparameters for the whole training.^[1] In RapidMiner, the Loop Parameters operator iterates through all parameter combinations as Optimize Parameters (Grid) does, but without optimising, which can be useful for plotting and logging.^[5] Some scikit-learn models allow specialised, efficient parameter search strategies as alternatives to brute-force search, such as model-specific cross-validation, information criteria and out-of-bag estimates.^[3]

Sources

1. [1]Hyperparameter optimization - Wikipediasecondary source
2. [2]Random Search for Hyper-Parameter Optimizationsecondary source
3. [3]3.2. Tuning the hyper-parameters of an estimator — scikit-learn 1.9.1 documentationdocumentation
4. [4]mlpack documentationdocumentation
5. [5]Optimize Parameters (Grid) - Rapidminer Documentationdocumentation