In the last chapter, we covered the ability of Convolutional Neural Networks (CNNs) to learn feature representations from grid-like data. In this chapter, we introduce recurrent neural networks (RNNs), which are designed for processing sequential data.
Feedforward Neural Networks (FFNNs) treat the feature vectors for each sample as independent and identically distributed. As a result, they do not systematically take prior data points into account when evaluating the current observation. In other words, they have no memory.
One-dimensional convolutions, which we covered in the previous chapter, produce sequence elements that are a function of a small number of their neighbors. However, they only allow for shallow parameter-sharing by applying the same convolutional kernel to the relevant time steps.
The major innovation of the RNN model is that each output...
