It helps to avoid local optimas when using smaller learning rates. However, it often takes longer to converge. What can help shorten the training time is using a warm-up period. In this period, we can use a bigger learning rate for the first few epochs. After a certain number of epochs, we can decrease the learning rate. It's even possible to decrease the learning rate after each step, but this is not recommended, because you might be better off using a different optimizer instead (for example, if you want to use decay, you can specify this in as a hyperparameter). In theory, when the learning rate is too big during the warm-up period, it can be the case that you won't be able to reach the global optima at all.
In the following recipe, we demonstrate how to set a custom learning rate scheduler with Keras.
