What do you get with Print?

Instant access to your digital eBook copy whilst your Print order is Shipped

Paperback book shipped to your preferred address

Download this book in EPUB and PDF formats

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

AI Assistant (beta) to help accelerate your learning

Up and Running with Reinforcement Learning

This book will cover interesting topics in deep Reinforcement Learning (RL), including the more widely used algorithms, and will also provide TensorFlow code to solve many challenging problems using deep RL algorithms. Some basic knowledge of RL will help you pick up the advanced topics covered in this book, but the topics will be explained in a simple language that machine learning practitioners can grasp. The language of choice for this book is Python, and the deep learning framework used is TensorFlow, and we expect you to have a reasonable understanding of the two. If not, there are several Packt books that cover these topics. We will cover several different RL algorithms, such as Deep Q-Network (DQN), Deep Deterministic Policy Gradient (DDPG), Trust Region Policy Optimization (TRPO), and Proximal Policy Optimization (PPO), to name a few. Let's dive right into deep RL.

In this chapter, we will delve deep into the basic concepts of RL. We will learn the meaning of the RL jargon, the mathematical relationships between them, and also how to use them in an RL setting to train an agent. These concepts will lay the foundations for us to learn RL algorithms in later chapters, along with how to apply them to train agents. Happy learning!

Some of the main topics that will be covered in this chapter are as follows:

Formulating the RL problem
Understanding what an agent and an environment are
Defining the Bellman equation
On-policy versus off-policy learning
Model-free versus model-based training

Why RL?

RL is a sub-field of machine learning where the learning is carried out by a trial-and-error approach. This differs from other machine learning strategies, such as the following:

Supervised learning: Where the goal is to learn to fit a model distribution that captures a given labeled data distribution
Unsupervised learning: Where the goal is to find inherent patterns in a given dataset, such as clustering

RL is a powerful learning approach, since you do not require labeled data, provided, of course, that you can master the learning-by-exploration approach used in RL.

While RL has been around for over three decades, the field has gained a new resurgence in recent years with the successful demonstration of the use of deep learning in RL to solve real-world tasks, wherein deep neural networks are used to make decisions. The coupling of RL with deep learning is typically referred to as deep RL, and is the main topic of discussion of this book.

Deep RL has been successfully applied by researchers to play video games, to drive cars autonomously, for industrial robots to pick up objects, for traders to make portfolio bets, by healthcare practitioners, and copious other examples. Recently, Google DeepMind built AlphaGo, a RL-based system that was able to play the game Go, and beat the champions of the game easily. OpenAI built another system to beat humans in the Dota video game. These examples demonstrate the real-world applications of RL. It is widely believed that this field has a very promising future, since you can train neural networks to make predictions without providing labeled data.

Now, let's delve into the formulation of the RL problem. We will compare how RL is similar in spirit to a child learning to walk.

Formulating the RL problem

The basic problem that is solved is training a model to make predictions of some pre-defined task without any labeled data. This is accomplished by a trial-and-error approach, akin to a baby learning to walk for the first time. A baby, curious to explore the world around them, first crawls out of their crib not knowing where to go nor what to do. Initially, they take small steps, make mistakes, keep falling on the floor, and cry. But, after many such episodes, they start to stand on their feet on their own, much to the delight of their parents. Then, with a giant leap of faith, they start to take slightly longer steps, slowly and cautiously. They still make mistakes, albeit fewer than before.

After many more such tries—and failures—they gain more confidence that enables them to take even longer steps. With time, these steps get much longer and faster, until eventually, they start to run. And that's how they grow up into a child. Was any labeled data provided to them that they used to learn to walk? No. they learned by trial and error, making mistakes along the way, learning from them, and getting better at it with infinitesimal gains made for every attempt. This is how RL works, learning by trial and error.

Building on the preceding example, here is another situation. Suppose you need to train a robot using trial and error, this is how to do it. Let the robot wander randomly in the environment initially. The good and bad actions are collected and a reward function is used to quantify them, thus, a good action performed in a state will have high rewards; on the other hand, bad actions will be penalized. This can be used as a learning signal for the robot to improve itself. After many such episodes of trial and error, the robot would have learned the best action to perform in a given state, based on the reward. This is how learning in RL works. But we will not talk about human characters for the rest of the book. The child described previously is the agent, and their surroundings are the environment in RL parlance. The agent interacts with the environment and, in the process, learns to undertake a task, for which the environment will provide a reward.

Key benefits

Explore efficient Reinforcement Learning algorithms and code them using TensorFlow and Python

Train Reinforcement Learning agents for problems, ranging from computer games to autonomous driving.

Formulate and devise selective algorithms and techniques in your applications in no time.

Description

Advances in reinforcement learning algorithms have made it possible to use them for optimal control in several different industrial applications. With this book, you will apply Reinforcement Learning to a range of problems, from computer games to autonomous driving. The book starts by introducing you to essential Reinforcement Learning concepts such as agents, environments, rewards, and advantage functions. You will also master the distinctions between on-policy and off-policy algorithms, as well as model-free and model-based algorithms. You will also learn about several Reinforcement Learning algorithms, such as SARSA, Deep Q-Networks (DQN), Deep Deterministic Policy Gradients (DDPG), Asynchronous Advantage Actor-Critic (A3C), Trust Region Policy Optimization (TRPO), and Proximal Policy Optimization (PPO). The book will also show you how to code these algorithms in TensorFlow and Python and apply them to solve computer games from OpenAI Gym. Finally, you will also learn how to train a car to drive autonomously in the Torcs racing car simulator. By the end of the book, you will be able to design, build, train, and evaluate feed-forward neural networks and convolutional neural networks. You will also have mastered coding state-of-the-art algorithms and also training agents for various control problems.

What you will learn

Understand the theory and concepts behind modern Reinforcement Learning algorithms

Code state-of-the-art Reinforcement Learning algorithms with discrete or continuous actions

Develop Reinforcement Learning algorithms and apply them to training agents to play computer games

Explore DQN, DDQN, and Dueling architectures to play Atari s Breakout using TensorFlow

Use A3C to play CartPole and LunarLander

Train an agent to drive a car autonomously in a simulator

What do you get with Print?

Instant access to your digital eBook copy whilst your Print order is Shipped

Paperback book shipped to your preferred address

Download this book in EPUB and PDF formats

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

AI Assistant (beta) to help accelerate your learning

Frequently bought together

TensorFlow Reinforcement Learning Quick Start Guide

Mex$553.99

Mex$676.99

Total Mex$ 1,230.98

Praveen Narayanan Jun 27, 2019

This book presents a readable, instructive overview of the latest RL methods for the beginning practitioner. It walks the reader through the subject with motivating examples and well chosen code to get their hands dirty.

Amazon Verified review

Colbert Philippe Nov 20, 2019

This is a fantastic book for those starting in the field.

TensorFlow Reinforcement Learning Quick Start Guide: Get up and running with training and deploying intelligent, self-learning agents using Python

What do you get with Print?

TensorFlow Reinforcement Learning Quick Start Guide

Up and Running with Reinforcement Learning

Why RL?

Formulating the RL problem

Page 1 of 13

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Product Details

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs

TensorFlow Reinforcement Learning Quick Start Guide: Get up and running with training and deploying intelligent, self-learning agents using Python

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs