The A3C algorithm

As we mentioned earlier, we have parallel workers in A3C, and each worker will compute the policy gradients and pass them on to the central (or master) processor. The A3C paper also uses the advantage function to reduce variance in the policy gradients. The loss functions consist of three losses, which are weighted and added; they include the value loss, the policy loss, and an entropy regularization term. The value loss, L_v, is an L2 loss of the state value and the target value, with the latter computed as a discounted sum of the rewards. The policy loss, L_p, is the product of the logarithm of the policy distribution and the advantage function, A. The entropy regularization, L_e, is the Shannon entropy, which is computed as the product of the policy distribution and its logarithm, with a minus sign included. The entropy regularization term is like a bonus for...