So far, we have initialized a learning rate, and it has remained the same across all the epochs while training the model. However, initially, it would be intuitive for the weights to be updated quickly to a near-optimal scenario. From then on, they should be updated very slowly since the amount of loss that gets reduced initially is high and the amount of loss that gets reduced in the later epochs would be low.
This calls for having a high learning rate initially and gradually lowering it later on as the model achieves near-optimal accuracy. This requires us to understand when the learning rate must be reduced.
One potential way we can solve this problem is by continually monitoring the validation loss and if the validation loss does not decrease (let's say, over the previous x epochs), then we reduce the learning rate.
PyTorch provides us with tools we can use to perform learning rate reduction when the validation loss does not...
