Inference with NMT
Inferencing is slightly different from the training process for NMT (Figure 10.11). As we do not have a target sentence at the inference time, we need a way to trigger the decoder at the end of the encoding phase. This shares similarities with the image captioning exercise we did in Chapter 9, Applications of LSTM – Image Caption Generation. In that exercise, we appended the <SOS> token to the beginning of the captions to denote the start of the caption and <EOS> to denote the end.
We can simply do this by giving <s> as the first input to the decoder, then by getting the prediction as the output, and by feeding in the last prediction as the next input to the NMT:
Preprocess xs as explained previously
-
Feed xs into
and calculate v conditioned on xs -
Initialize
with v -
For the initial prediction step, predict
by conditioning the prediction on 
and v -
For subsequent time steps, while
, predict 
by conditioning the prediction on 
and v
Figure 10.11: Inferring...
