Chapter 1, Up and Running with Reinforcement Learning, provides an overview of the basic concepts of RL, such as an agent, an environment, and the relationship between them. It also covers topics such as reward functions, discounted rewards, and value and advantage functions. The reader will also get familiar with the Bellman equation, on-policy and off-policy algorithms, as well as model-free and model-based RL algorithms.
Chapter 2, Temporal Difference, SARSA, and Q-learning, introduces the reader to temporal difference learning, SARSA, and Q-learning. It also summarizes how to code these algorithms in Python, and to train and test them on two classical RL problems – GridWorld and Cliff Walking.
Chapter 3, Deep Q-Network, introduces the reader to the first deep RL algorithm of the book, DQN. It will also discuss how to code this in Python and TensorFlow. The code will then be used to train an RL agent to play Atari Breakout.
Chapter 4, Double DQN, Dueling Architectures, and Rainbow, builds on the previous chapter and extends it to double DQN. It also discusses dueling network architectures that involve value and advantage streams. These extensions will be coded in Python and TensorFlow, and will be used to train RL agents to play Atari Breakout. Finally, Google's dopamine code will be introduced and will be used to train a Rainbow DQN agent.
Chapter 5, Deep Deterministic Policy Gradient, is the first actor-critic algorithm of the book as well as the first RL algorithm for continuous control. It introduces policy gradients to the reader and discusses how to use it to train the policy for the actor. The chapter will code this algorithm using Python and TensorFlow and use it to train an agent to play the inverted pendulum problem.
Chapter 6, Asynchronous Methods – A3C and A2C, introduces the reader to the A3C algorithm, which is an asynchronous RL algorithm where one master processor will update the policy network, and multiple worker processors will use it to collect experience samples, which will be used to compute the policy gradients, and then passed on to the master processor. Also in this chapter, A3C will be used to train RL agents to play OpenAI Gym's CartPole and LunarLander. Finally, A2C is also briefly introduced.
Chapter 7, Trust Region Policy Optimization and Proximal Policy Optimization, discusses two RL algorithms based on the policy distribution ratio—TRPO and PPO. This chapter also discusses how to code PPO using Python and TensorFlow, and will use it to train an RL agent to solve the MountainCar problem in OpenAI Gym.
Chapter 8, Deep RL Applied to Autonomous Driving, introduces the reader to the TORCS racing car simulator, coding the DDPG algorithm for training an agent to drive a car autonomously. The code files for this chapter also include the PPO algorithm for the same TORCS problem, and is provided as an exercise for the reader.
