Working principles of RNNs
In this section, we will first provide some contextual information about RNNs. Then we will see some potential drawbacks of the classical RNN. Finally, we will see an improved variation of RNNs called LSTM to address the drawbacks.
Human beings do not start thinking from scratch. The human mind has so-called persistence of memory, the ability to associate the past with recent information. Traditional neural networks instead ignore past events. Take the movie scenes classifier as an example; it is not possible for a neural network to use past scenes to classify current ones. RNNs were developed to try to solve this problem.
Figure 1: RNNs have loops
In contrast to conventional neural networks, RNNs are networks with a loop that allows the information to be persistent in a neural network. In the preceding diagram, with the network A, at some time t, it receives the input 
and outputs a value of 
. So, in the preceding figure, we think of an RNN as multiple copies of...
