Introduction to RNNs

RNNs are neural networks that can process sequential data with a variable length. Examples of such data include the words of a sentence or the price of stock at various moments in time. By using the word sequential, we imply that the elements of the sequence are related to each other and that their order matters. For example, if we take a book and randomly shuffle all of the words in it, the text will lose its meaning, even though we'll still know the individual words. Naturally, we can use RNNs to solve tasks that relate to sequential data. Examples of such tasks are language translation, speech recognition, predicting the next element of a time series, and so on.

RNNs get their name because they apply the same function over a sequence recurrently. We can define an RNN as a recurrence relation:

Here, f is a differentiable function, s_t is a vector of...