Trust Region Policy Optimization and Proximal Policy Optimization

In the last chapter, we saw the use of A3C and A2C, with the former being asynchronous and the latter synchronous. In this chapter, we will see another on-policy reinforcement learning (RL) algorithm; two algorithms, to be precise, with a lot of similarities in the mathematics, differing, however, in how they are solved. We will be introduced to the algorithm called Trust Region Policy Optimization (TRPO), which was introduced in 2015 by researchers at OpenAI and the University of California, Berkeley (the latter is incidentally my former employer!). This algorithm, however, is difficult to solve mathematically, as it involves the conjugate gradient algorithm, which is relatively difficult to solve; note that first order optimization methods, such as the well established Adam and Stochastic Gradient Descent (SGD...