Values and policy
Before getting to the main subject of this chapter, policy gradients, let’s refresh our minds with the common characteristics of the methods covered in Part 2 of this book. The central topic in value iteration and Q-learning is the value of the state (V s) or value of the state and action (Qs,a). Value is defined as the discounted total reward that we can gather from this state or by issuing this particular action from the state. If we know this quantity, our decision on every step becomes simple and obvious: we just act greedily in terms of value, and that guarantees us a good total reward at the end of the episode. So, the values of states (in the case of the value iteration method) or state + action (in the case of Q-learning) stand between us and the best reward. To obtain these values, we have used the Bellman equation, which expresses the value in the current step via the value in the next step.
In Chapter 1, we defined the entity that...
