Tuning the loss function

While training a neural network for a learning problem, the objective of the network is to minimize the loss function. The loss function — also known as error, cost function, or opimization function–compares the prediction with the ground truth during the forward pass. The output of this loss function is used to optimize the weights during the backward pass. Therefore, the loss function is crucial in training the network. By setting the correct loss function, we force the network to optimize towards the desired predictions. For example, for imbalanced datasets we need a different loss function.

In the previous recipes, we've used mean squared error (MSE) and categorical entropy as loss functions. There are also other popular loss functions, and another option is to create a custom loss function. A custom loss function gives the ability to optimize to the desired output. This will be important when we will implement Generative Adversarial Networks (GANs). In the...