An RL agent is an optimization process that learns from experience, using data from its environment that it has collected through its own observations. It starts out knowing nothing about a task explicitly, learns by trial and error about what happens when it makes decisions, keeps track of successful decisions, and makes those same decisions under the same circumstances in the future.

In fields other than AI, RL is also referred to as dynamic programming. It takes much of its basic operating structure from behavioral psychology, and many of its mathematical constructs such as utility functions are taken from fields such as economics and game theory.

Let's get familiar with some key concepts in RL:

• Agent: This is the decision-making entity.
• Environment: This is the world in which the agent operates, such as a game to win or task to accomplish.
• State: This...

What does it mean to be in a state, to take an action, or to receive a reward? These are the most important concepts for us to understand intuitively, so let's dig deeper into them. The following diagram depicts the agent-environment interaction in an MDP:

The agent interacts with the environment through actions, and it receives rewards and state information from the environment. In other words, the states and rewards are feedback from the environment, and the actions are inputs to the environment from the agent.

Going back to our simple driving simulator example, our agent might be moving or stopped at a red light, turning left or right, or heading straight. There might be other cars in the intersection, or there might not be. Our distance from the destination will be X units.

Here, we'll go over some of the most important concepts that we'll need to bear in mind throughout our study of RL. We'll focus heavily on topics that are specific to Q-learning, but we'll also explore topics relating to other branches of RL, such as the related algorithm SARSA and policy-based RL algorithms.

We'll be using value-based iteration for the projects in this book. The description of the Bellman equation given previously offers a very high-level understanding of how value-based iteration works. The main difference is that in value-based iteration, the agent learns the expected reward value of each state-action pair, and in policy...

Similar to Q-learning, SARSA is a model-free RL method that does not explicitly learn the agent's policy function.

The primary difference between SARSA and Q-learning is that SARSA is an on-policy method while Q-learning is an off-policy method. The effective difference between the two algorithms happens in the step where the Q-table is updated. Let's discuss what that means with some examples:

Monte Carlo tree search (MCTS) is a type of model-based RL. We won't be discussing it in detail here, but it's useful to explore further as a contrast to model-free RL algorithms. Briefly, in model-based RL, we attempt to explicitly model a value function instead of relying on sampling and observation, so that we don't have to rely as much on trial and error in the learning process.

RL is one of the most exciting and fastest-growing branches of machine learning, with the greatest potential to create powerful optimization solutions to wide-ranging computing problems. As we have seen, Q-learning is one of the most accessible branches of RL and will provide a beginning RL practitioner and experienced programmer a strong foundation for developing solutions to both straightforward and complex optimization problems.

In the next chapter, we'll learn about Q-learning in detail, as well as about the learning agent that we'll be training to solve our Q-learning task. We'll discuss how Q-learning solves MDPs using a state-action model and how to apply that to our programming task.

1. What is the difference between a reward and a value?
2. What is a hyperparameter? Give an example of a hyperparameter other than the ones discussed in this chapter.
3. Why will a Q-learning agent not choose the highest Q-valued action for its current state?
4. Explain one benefit of a decaying gamma.
5. Describe in one or two sentences the difference between the decision-making strategies of SARSA and Q-learning.
6. What kind of policy does Q-learning implicitly assume the agent is following?
7. Under what circumstances will SARSA and Q-learning produce the same results?