Recurrent Neural Networks

Just as humans do not reset their thinking every second, neural networks that aim to understand human language should not do so either. This means that in order to understand each word from a paragraph or even a whole book, you or the model need to understand the previous words, which can help give context to words that may have different meanings.

Traditional neural networks, as we have discussed so far, are not capable of performing such tasks – hence the creation of the concept and network architecture of RNNs. As we briefly explained previously, these network architectures contain loops among the different nodes. This allows information to remain in the model for longer periods of time. Due to this, the output from the model becomes both a prediction and a memory, which will be used when the next bit of sequenced text is passed through the model.

This concept goes back to the 1980s, although it has only become popular recently thanks to advances...