Train a neural network

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

Neural networks typically require vast numbers of sample inputs to achieve a given level of function.^[1] Training a neural network is a compute-intensive process, accelerated by graphics processing units (GPUs) and large datasets.^[1] On an ordinary table of rows and columns, fit a simpler model first, and keep the neural network only if it beats that model on rows it was not trained on. (Meridian's view)

A hyperparameter is any configurable part of the network and its learning process whose value is set before training, such as the learning rate, the batch size or the number of nodes and layers.^[1] A network must avoid over-training when it has to generalise to unseen examples, and over-training arises when the network's capacity is much larger than needed.^[1] Cross-validation and similar techniques can check for over-training and select hyperparameters that minimise generalisation error.^[1] Training a neural network may not consistently converge on a single solution, because the system may get stuck in a local minimum.^[1]

In RapidMiner, the Deep Learning operator is deterministic only when its reproducible parameter is set to true, and it then uses a single thread.^[2]

RapidMiner's Neural Net operator uses a sigmoid as its activation function, the non-linear function by which each neuron computes its output from its inputs, so attribute values should be scaled to between -1 and +1.^[3][1] In Alteryx's Neural Network tool, the numeric methods that optimise the model's weights can be problematic when the predictor fields are on different scales, such as income beside the number of household members.^[4] Scale every numeric column before training, and keep the scaling with the model so that new rows are scaled the same way. (Meridian's view)

Neural networks are black box systems whose decision-making processes are complicated to understand.^[1] A neural network depends on the quality of its training data, and biased data produces biased results.^[1] Where someone has to explain each prediction, prefer a model whose working can be read. (Meridian's view)

The method

In a neural network, artificial neurons are connected by edges, and the strength of the signal at each connection is determined by a weight, which adjusts as part of the training process.^[1] Neurons are grouped into layers, and signals travel from the input layer to the output layer, typically passing through intermediate hidden layers.^[1] A network is typically called a deep neural network if it has at least two hidden layers.^[1] A multilayer perceptron (MLP) is a feedforward neural network of fully connected neurons with nonlinear activation functions, organised in layers, in which information flows from input to output with no cycles.^[5]

A loss function measures the discrepancy between a model's predictions and the desired values, usually as a non-negative number where a perfect prediction incurs a loss of 0.^[1] In regression the common choices of loss are mean squared error and mean absolute error, and in classification the standard choice is cross-entropy.^[1] Backpropagation spreads the error from the output nodes across the network to the input nodes.^[1] Technically, backpropagation calculates the gradient of the loss function with respect to the weights.^[1] In stochastic learning the weights are adjusted after each sample, in batch learning after a batch of inputs, and a common compromise is mini-batches selected stochastically from the whole dataset.^[1] The learning rate defines the size of the corrective steps the model takes, and a high rate shortens training but with lower ultimate accuracy.^[1] Momentum makes each weight adjustment depend to some degree on the previous change.^[1] In RapidMiner, the Deep Learning operator trains a multi-layer feed-forward network with stochastic gradient descent using back-propagation.^[2]

Resilient backpropagation (Rprop), created by Martin Riedmiller and Heinrich Braun in 1992, takes into account only the sign of each partial derivative, not its magnitude, and acts independently on each weight.^[6] AutoMLP, an algorithm by Thomas Breuel and Faisal Shafait, trains a small ensemble of networks with different learning rates and numbers of hidden units, and replaces the worst performers with modified copies of the best.^[7] In KNIME, the PNN Learner (DDA) trains a probabilistic neural network (PNN) by Dynamic Decay Adjustment (DDA), and the network's rules are high-dimensional Gaussian functions, each with a centre vector and a standard deviation adjusted during training.^[8]

In 1958, the psychologist Frank Rosenblatt described the perceptron, one of the first implemented neural networks.^[1] Backpropagation was developed independently several times in the early 1970s, the earliest instance being Seppo Linnainmaa's master's thesis of 1970.^[5] In 1986, David E. Rumelhart and colleagues popularised backpropagation.^[5][1]

Compared with the alternatives

The simplest feedforward neural network is a linear network of a single layer of output nodes, and fitting its weights by minimising mean squared error is the method of least squares, or linear regression.^[1] The MLP grew out of an effort to improve on single-layer perceptrons, which could only be applied to linearly separable data.^[5] A neural network earns its cost where the pattern is too tangled for a linear model and there are enough rows to learn it; otherwise a linear regression or a boosted or forest model is the easier one to trust. (Meridian's view)

Sources

1. [1]Neural network (machine learning) - Wikipediasecondary source
2. [2]Deep Learning - Rapidminer Documentationdocumentation
3. [3]Neural Net - Rapidminer Documentationdocumentation
4. [4]Neural Network Tooldocumentation
5. [5]Multilayer perceptron - Wikipediasecondary source
6. [6]Rprop - Wikipediasecondary source
7. [7]AutoMLP - Rapidminer Documentationdocumentation
8. [8]PNN Learner (DDA)documentation