Chapter 7

1. Trust Region Policy Optimization (TRPO) has an objective function and a constraint. It hence requires a second order optimization such as a conjugate gradient. SGD and Adam are not applicable in TRPO.
2. The entropy term helps in regularization. It allows the agent to explore more.
3. We clip the policy ratio to limit the amount by which one update step will change the policy. If this clipping parameter epsilon is large, the policy can change drastically in each update, which can result in a sub-optimal policy, as the agent's policy is noisier and has too many fluctuations.
4. The action is bounded between a negative and a positive value, and so the tanh activation function is used for mu. For sigma, the softplus is used as sigma and is always positive. The tanh function cannot be used for sigma, as tanh can result in negative values for sigma, which is meaningless!
5. Reward...