Another drawback of recurrent neural networks is the problem of exploding gradients. This is similar to the vanishing gradient problem but the exact opposite. Sometimes, during backpropagation, the gradients explode to extraordinarily large values. As with the vanishing gradient problem, the problem of exploding gradients occurs when network architectures get deeper.
Pain point #2 – The exploding gradient problem
Getting ready
The name exploding gradient problem stems from the fact that, during the backpropagation step, some of the gradients vanish or become zero. Technically, this means that there is no error term being propagated backward during the backward pass of the network. This becomes a problem when...
