Training the NMT

Now that we have defined the NMT architecture and preprocessed training data, it is quite straightforward to train the model. Here we will define and illustrate (see Figure 10.10) the exact process used for training:

1. Preprocess

   

   as explained previously

2. Feed x_s into the and calculate v conditioned on x_s
3. Initialize with v
4. Predict corresponding to the input sentence x_s from , where the m_th prediction, out of the target vocabulary V is calculated as follows:
   
   

   Here, wTm denotes the best target word for mth position.

5. Calculate the loss: categorical cross-entropy between the predicted word, , and the actual word at the position,
6. Optimize both the , , and softmax layer with respect to the loss
   

   Figure 10.10: The training procedure for the NMT