A loss function (that is, an error measurement) is a necessary part of the training of an ANN. It is a measure of the extent to which the calculated output of a network during training differs from its required output. By differentiating the loss function, we can find a quantity with which to adjust the weights of the connections between the layers so as to make the calculated output of the ANN more closely match the required output.
The simplest loss function is the mean squared error:
,
Here, y is the actual label value, and 
is the predicted label value.
Of particular note is the categorical cross-entropy loss function, which is given by the following equation:
This loss function is used when only one class is correct out of all the possible ones and so is used when the softmax function is used as the output of the final layer of an ANN.
Note that both of these...
