RNNs

Recurrent often means occurring repeatedly. The recurrent part of RNNs simply means that the same task is done over all the inputs in the input sequence (for RNNs, we give a sequence of timesteps as the input sequence). One main difference between feed forward networks and RNNs is that RNNs have memory elements called states that capture the information from the previous inputs. So, in this architecture, the current output not only depends on the current input, but also on the current state, which takes into account past inputs.

RNNs are trained by sequences of inputs rather than a single input; similarly, we can consider each input to an RNN as a sequence of timesteps. The state elements in RNNs contain information about past inputs to process the current input sequence.

Figure 5.3: RNN structure

For each input in the input sequence, the RNN gets a state, calculates its output, and sends its state to the next input in the sequence. The same set of tasks is repeated for...