Introduction

Of all the machine-learning algorithms we have considered thus far, none have considered data as a sequence.To take sequence data into account, we extend neural networks that store outputs from prior iterations. This type of neural network is called a recurrent neural network (RNN).Consider the fully connected network formulation:

Here, the weights are given by Amultiplied by the input layer, x, and then run through an activation function, , which gives the output layer, y.If we have a sequence of input data, , we can adapt the fully connected layer to take prior inputs into account, as follows:

On top of this recurrent iteration to get the next input, we want to get the probability distribution output, as follows:

Once we have a full sequence output, , we can consider the target a number or category by just considering the last output.See the following figure for how a general architecture might work:

Figure 1: To predict a single number, or a category, we take a sequence of inputs...