The Bellman equation of optimality

To explain the Bellman equation, it’s better to go a bit abstract. Don’t be afraid; I’ll provide concrete examples later to support your learning! Let’s start with a deterministic case, when all our actions have a 100% guaranteed outcome. Imagine that our agent observes state s₀ and has N available actions. Every action leads to another state, s₁…s_N, with a respective reward, r₁…r_N. Also, assume that we know the values, V _i, of all states connected to state s₀. What will be the best course of action that the agent can take in such a state?

Figure 5.3: An abstract environment with N states reachable from the initial state

If we choose the concrete action, a_i, and calculate the value given to this action, then the value will be V ₀(a = a_i) = r_i + V _i. So, to choose the best possible action, the agent needs to calculate the resulting values for every action and...