Computing activations per timestep

As we previously pointed out in the LSTM architecture, it is fed the memory and activation values from the previous timestep separately. This is distinctly separate from the assumption we made with the GRU unit, where at = ct. This dual manner of data processing is what lets us conserve relevant representations in memory across very long sequences, potentially even 1,000 timesteps! The activations are, however, always functionally related to the memory (c^t) at each time step. So, we can compute the activations at a given timestep by first applying a tanh function to the memory (c^t), then performing an element-wise computation of the result with the output gate value (Γo). Note that we do not initialize a weight matrix at this step, but simply apply tanh to each element in the (c^t) vector. This can be mathematically represented as follows...