Improving NMTs
As you can see from the preceding results, our translation model is not behaving ideally. These results were obtained by running the optimization for more than 12 hours on a single NVIDIA 1080 Ti GPU. Also note that this is not even the full dataset, we only used 250,000 sentence pairs for training. However, if you type something into Google Translate, which uses the Google Neural Machine Translation (GNMT) system, the translation almost always looks very realistic with only minor mistakes. So it is important to know how we can improve the model so that it can produce better results. In this section, we will discuss several ways of improving NMTs such as teacher forcing, deep LSTMs, and attention mechanism.
Teacher forcing
As we discussed in the Training the NMT section, we do the following to train the NMT:
First, we fed the full encoder sentence to obtain the final state outputs of the encoder
We then set the final states of the encoder to be the initial state of the decoder...
