Summary

AI and deep learning are making huge advances in terms of images and artificial vision thanks to convolutional networks. But RNNs also have a lot of power.

In this chapter, we reviewed how a neural network would can to predict the values of a sine function using temporal sequences. If you change the training data, this architecture can learn about stock movements for each distribution. Also, there are many architectures for RNNs, each of which is optimized for a certain task. But RNNs have a problem with vanishing gradients. A solution to this problem is a new model, called LSTM, which changes the structure of a neuron to memorize timesteps.

Focusing on linguistics, statistical LMs have many problems related with computational load and distribution probabilities. To solve the sparsity problem, the size of the n-gram model was lowered to 4 or 3 grams, but that was an insufficient number of steps back to predict an upcoming word. If we use this approach, the sparsity problem appears...