Knowledge of the following will help you to better understand the concepts presented in this chapter:

• Python (version 2 or 3)
• NumPy
• TensorFlow (version 1.4 or higher)

We will first learn about TD learning. This is a very fundamental concept in RL. In TD learning, the learning of the agent is attained by experience. Several trial episodes are undertaken of the environment, and the rewards accrued are used to update the value functions. Specifically, the agent will keep an update of the state-action value functions as it experiences new states/actions. The Bellman equation is used to update this state-action value function, and the goal is to minimize the TD error. This essentially means the agent is reducing its uncertainty of which action is the optimal action in a given state; it gains confidence on the optimal action in a given state by lowering the TD error.

In this section, we will learn about SARSA and Q-Learning and how can they are coded with Python. Before we go further, let's find out what SARSA and Q-Learning are. SARSA is an algorithm that uses the state-action Q values to update. These concepts are derived from the computer science field of dynamic programming, while Q-learning is an off-policy algorithm that was first proposed by Christopher Watkins in 1989, and is a widely used RL algorithm.

SARSA is another on-policy algorithm that was very popular, particularly in the 1990s. It is an extension of TD-learning, which we saw previously, and is an on-policy algorithm. SARSA keeps an update of the state-action value...

Let's consider cliff walking and grid world problems. First, we will introduce these problems to you, then we will proceed on to the coding part. For both problems, we consider a rectangular grid with nrows (number of rows) and ncols (number of columns). We start from one cell to the south of the bottom left cell, and the goal is to reach the destination, which is one cell to the south of the bottom right cell.

Note that the start and destination cells are not part of the nrows x ncols grid of cells. For the cliff walking problem, the cells to the south of the bottom row of cells, except for the start and destination cells, form a cliff where, if the agent enters, the episode ends with catastrophic fall into the cliff. Likewise, if the agent tries to leave the left, top, or right boundaries of the grid of cells, it is placed back in the...

In this chapter, we looked at the concept of TD. We also learned about our first two RL algorithms: Q-learning and SARSA. We saw how you can code these two algorithms in Python and use them to solve the cliff walking and grid world problems. These two algorithms give us a good understanding of the basics of RL and how to transition from theory to code. These two algorithms were very popular in the 1990s and early 2000s, before deep RL gained prominence. Despite that, Q-learning and SARSA still find use in the RL community today.

In the next chapter, we will look at the use of deep neural networks in RL that gives rise to deep RL. We will see a variant of Q-learning called Deep Q-Networks (DQNs) that will use a neural network instead of a tabular state-action value function, which we saw in this chapter. Note that only problems with small number of states and actions are...

• Reinforcement Learning: an Introduction by Richard Sutton and Andrew Barto, 2018