Bidirectional RNNs
The RNN models we've just looked at – LSTMs, GRUs – are powerful indeed and provide extremely good results when it comes to sequence-processing tasks. Now, let's discuss how to make them even more powerful, and the methods that yield the amazing successes in deep learning that you have been hearing about.
Let's begin with the idea of bidirectional RNNs. The idea applies to all variants of RNNs, including, but not limited to, LSTMs and GRUs. Bidirectional RNNs process the sequence in both directions, allowing the network to have both backward and forward information about the sequence, providing it with a much richer context:
The bidirectional model essentially employs two RNNs in parallel – one as the "forward layer" and the other as the "backward layer". As shown in the preceding figure, the forward layer processes the sequence in the order of its...