Simple Recurrent Neural Network

Here is what a simple neural network with loops looks like:

RNN Network

In this diagram, a Neural Network N takes input to produce output . Due to the loop, at the next time step , it takes the input along with input to produce output . Mathematically, we represent this as the following equation:

When we unroll the loop, the RNN architecture looks as follows at time step :

Unrolled RNN at timestep t₁

As the time steps evolve, this loop unrolls as follows at time step 5:

Unrolled RNN at timestep t₅

At every time step, the same learning function, , and the same parameters, w and b, are used.

The output y is not always produced at every time step. Instead, an output h is produced at every time step, and another activation function is applied to this output h to produce the output y. The equations for the RNN look like this now:

where,

...