Bi-directional RNNs

This section of the chapter will discuss the major limitations of RNNs and how bi-directional RNN, a special type of RNN helps to overcome those shortfalls. Bi-directional neural networks, apart from taking inputs from the past, takes the information from the future context for its required prediction.

Shortfalls of RNNs

The computation power of standard or unidirectional RNNs has constraints, as the current state cannot reach its future input information. In many cases, the future input information coming up later becomes extremely useful for sequence prediction. For example, in speech recognition, due to linguistic dependencies, the appropriate interpretation of the voice as a phoneme might depend on the next few spoken words. The same situation might also arise in handwriting recognition.

In some modified versions of RNN, this feature is partially attained by inserting some delay of a certain amount (N) of time steps in the output. This delay helps to capture the future...