Popular architecture

Now that we have learned about all the components that are used to contrast RNNs, let's explore a popular architecture that has been developed by researchers in the field—the clockwork RNN (CW-RNN).

Clockwork RNNs

As we have learned, it is very challenging to discover long-term dependencies in RNNs, and LSTMs and GRUs were designed to overcome this limitation. CW-RNN, created by a group at IDSIA led by Jürgen Schmidhuber, modifies the vanilla RNN module such that the hidden layer is partitioned into separate modules, each of which processes its inputs at different temporal granularities. What this means is that the hidden layers perform computations at their preset clock rate (which is...