Building a better optimizer
In this chapter we have so far discussed several examples in which better neural network architectures allowed for breakthroughs; however, just as (and perhaps even more) important is the optimization procedure used to minimize the error function in these problems, which "learns" the parameters of the network by selecting those that yield the lowest error. Referring to our discussion of backpropagation, this problem has two components:
- How to initialize the weights: In many applications historically, we see that the authors used random weights within some range, and hoped that the use of backpropagation would result in at least a locally minimal loss function from this random starting point. Whether the activation functions in the network had staturated or 0 values (increasing the likelihood of uninformative gradients during training of the model) was not considered.
- How to find the local minimum loss: In basic backpropagation, we used gradient descent...
