Chapter 6

1. Asynchronous Advantage Actor-Critic Agents (A3C) is an on-policy algorithm, as we do not use a replay buffer to sample data from. However, a temporary buffer is used to collect immediate samples, which are used to train once, after which the buffer is emptied.
2. The Shannon entropy term is used as a regularizer—the higher the entropy, the better the policy is.
3. When too many worker threads are used, the training can slow down and can crash, as memory is limited. If, however, you have access to a large cluster of processors, then using a large number of worker threads/processes helps.
4. Softmax is used in the policy network to obtain probabilities of different actions.
5. An advantage function is widely used, as it decreases the variance of the policy gradient. Section 3 of the A3C paper (https://arxiv.org/pdf/1602.01783.pdf) has more regarding this.
6. This is an exercise...