Recurrent neural networks

RNNs get their name because they recurrently apply the same function over a sequence. An RNN can be written as a recurrence relation defined by this function:

S_t = f(S^t-1, X_t)

Here S_t —the state at step t—is computed by the function f from the state in the previous step, that is t-1, and an input X_t at the current step. This recurrence relation defines how the state evolves step by step over the sequence via a feedback loop over previous states, as illustrated in the following figure:

Figure from [3]

Left: Visual illustration of the RNN recurrence relation: S _t = S _t-1 * W + X _t * U. The final output will be o _t = V*S _t

Right: RNN states recurrently unfolded over the sequence t- 1, t, t+1. Note that the parameters U, V, and W are shared between all the steps.

Here f can be any differentiable function. For example, a basic RNN is defined by the following recurrence relation:

S_t = tanh(S_t-1 * W + X_t * U)

Here W defines a linear transformation from state to state...