Code repository, setup, and configuration

First of all, let's make sure you have all the information to access the code repository for this book. The source code provides you with all the necessary code samples that we will discuss in this book and provides additional details on how to set up and run the training or testing scripts for each chapter specifically. To get started, head to the book's code repository on GitHub at the following link: https://github.com/PacktPublishing/Hands-On-Intelligent-Agents-with-OpenAI-Gym.

Create a GitHub account if you do not already have one and fork the repository so that it is added to your own GitHub account. This is recommended as it allows you to make any changes to the code you prefer while following along, and also allow you to send a pull request when you have something cool to show and be featured on the book's blog!

You can clone the repository to a folder named HOIAWOG in your home directory using the following command:

git clone https://github.com/PacktPublishing/Hands-On-Intelligent-Agents-with-OpenAI-Gym.git ~/HOIAWOG

Note that the book assumes that you have set up the code repository at this particular location: ~/HOIAWOG. If you happen to change it for some reason, be sure to remember it and change some of the commands in the book accordingly.

If you are wondering why the directory name was chosen to be HOIAWOG, do not think anymore. It is an acronym for this book's title: Hands On Intelligent Agents With OpenAI Gym (HOIAWOG)!

The book's code repository will be kept up to date to take care of any changes in the external libraries or other software, so that the intelligent agent implementation code and other code samples are functional. Occasionally, new code and updates will also be added to help you explore developing intelligent agents further. To stay on top of the changes and be notified of updates, it is recommended you star the book's code repository from your GitHub account.

Toward the end of Chapter 1, Introduction to Intelligent Agents and Learning Environments, we did a quick install of OpenAI Gym to get a sneak peak into the Gym. That was a minimal install, to get us started quickly. In the next section, we will go over the installation step by step and make sure everything you need to develop agents using the Gym is installed and configured properly. We will go over the different levels and methods of installation here so that you are aware of the installation process in general. You may end up modifying your system, or using another system at home or work, or changing your computer altogether. This section will make sure that you can get everything set up in the right way. Feel free to pick the installation method that is suitable for your use cases.

Prerequisites

The only main prerequisite for using OpenAI Gym is Python 3.5+. To make further development easy and organized, we will use the Anaconda Python distribution. For those of you who are not familiar with Anaconda, it is a Python distribution (although a distribution for the R language is also available) that includes hundreds of popular machine learning and data science packages and comes with an easy-to-use package and virtual environment manager called conda. The good thing is that the Anaconda Python distribution is available for Linux, macOS, and Windows! Another main reason to use the Anaconda distribution is that it helps in easily creating, installing, managing, and upgrading an isolated Python virtual environment. This makes sure the code we learn about and develop in this book produces the same results, irrespective of the operating system we use. This will relieve you from solving dependency issues or library version mismatch issues that you would have had to handle manually if you were not using a Python distribution such as Anaconda. You will find that it just works, which is nice and cool. Let's get started and install the Anaconda Python distribution.

Open a command prompt or Terminal and enter the following:

praveen@ubuntu:~$wget http://repo.continuum.io/archive/Anaconda3-4.3.0-Linux-x86_64.sh -O ~/anaconda.sh

This command uses the wget tool to fetch/download the installation script for Anaconda version 3-4.3 and saves it as anaconda.sh in your home directory. This command should work on macOS and Linux (Ubuntu, Kubuntu, and so on), which come with the wget tool pre-installed. Note that we are downloading a specific version of Anaconda (3-4.3). This will make sure we have the same configuration throughout this book. Do not worry if this is not the latest version available. You can always upgrade the distribution later using this command:

conda update conda

anaconda.sh is a shell script that has all the things that are needed to install Anaconda on your system! If you are interested, you can open it using your favorite text editor to see how cleverly the binaries, the installation process instructions, and the shell commands have been all lumped together into a single file.

Let's now install the Anaconda Python distribution under your home directory. The following installation process is carefully laid out to make sure it works both on Linux and macOS systems. Before you enter the command, you should be aware of one thing. The following command will run the installer in silent mode. This means that it will use the default installation parameters and go ahead with the installation, without asking you yes/no for each and every configuration. This also means that you agree to the Anaconda distribution's licensing terms. In case you want to manually go through the installation process step by step, run the following command without the arguments -b and -f:

praveen@ubuntu:~$bash ~/anaconda.sh -b -f -p $HOME/anaconda

Wait for the installation process to complete and then we are done!

To start using conda and the other goodness in the Anaconda Python distribution, we should make sure that your system knows where to find the Anaconda tools. Let's add the Anaconda binaries directory by appending its path to the PATH environment variable, as shown here:

praveen@ubuntu:~$export PATH=$HOME/anaconda/bin:$PATH

I strongly advise you to add this line to the end of your ~/.bashrc file so that whenever you open a new bash terminal, the Anaconda tools are accessible.

You can type the following command to make sure the installation was successful:

praveen@ubuntu:~$conda list

This command will just print the list of packages available in your default environment.

Creating the conda environment

Now that we have set up Anaconda, let's use conda to create a Python virtual environment, which we will use throughout this book.

If you prefer a one-click install setup and do not want to go through the installation step by step, a greatly simplified way to create the environment with all the necessary packages installed is using the conda_env.yaml conda environment configuration file available in the book's code repository. You can simply run the following command from the book's code repository directory (HOIAWOG) which we created in the previous section:
praveen@ubuntu:~/HOIAWOG$ conda create -f conda_env.yaml -n rl_gym_book

At this point, we will just create a new minimal environment to proceed. Enter the following command in a Terminal:

praveen@ubuntu:~$conda create --name rl_gym_book python=3.5

This will create a conda environment named rl_gym_book with a Python3 interpreter. It will print some information about what is going to be downloaded and the packages that will be installed. You may be prompted with a yes/no question as to whether you want to proceed. Type y and hit Enter. Once the environment creation process is complete, you can activate that environment using the following command:

praveen@ubuntu:~$source activate rl_gym_book

You will now see your command prompt's prefix changing to look something like this, to signify that you are inside the rl_gym_book virtual environment:

(rl_gym_book) praveen@ubuntu:~$

You can use this as an indicator as you progress through the chapters to know when commands have to be entered inside this environment and when commands can be entered outside the environment. To exit or deactivate the environment, you can simply type this:

praveen@ubuntu:~$source deactivate

Minimal install – the quick and easy way

The OpenAI Gym is a Python package and is available in the Python Package Index (PyPI) repository. You can use easy_install or pip to fetch and install packages from the PyPI repository. Pip is a package management tool for Python, which most of you might be familiar with if you have experience scripting in Python:

(rl_gym_book) praveen@ubuntu:~$pip install gym

That's it!

Let's quickly check the installation actually went fine by running the following code. Create a gym_install_test.py file under the ~/rl_gym_book directory, type/copy the following code into it, and save it. You can also download the gym_quick_install_test.py file from the book's code repository:

#! /usr/bin/env python  
import gym
env = gym.make("MountainCar-v0") # Create a MountainCar environment
env.reset()
for _ in range(2000): # Run for 2000 steps
    env.render()
    env.step(env.action_space.sample()) # Send a random action

Let's try running the script:

(rl_gym_book) praveen@ubuntu:~/HOIAWOG$python gym_quick_install_test.py

This should pop up a new window showing a car/carton and a v-shaped mountain, and you should see the car moving left and right randomly. The mountain car window should look something like this screenshot:

You will also see some values printed out to the console/Terminal that look like this:

If you saw this happening, then rejoice! You now have a (minimal) setup of OpenAI Gym!

Complete install of OpenAI Gym learning environments

Not all environments are usable with the minimal installation. To be able to use most or all the environments available in the Gym, we will go through the installation of the dependencies and build OpenAI Gym from the latest source code on the master branch.

To get started, we will need to install the required system packages first. Next, you will find instructions for both Ubuntu and macOS. Choose the set of instructions based on your development platform.

Instructions for Ubuntu

The following commands were tested on Ubuntu 14.04 LTS and on Ubuntu 16.04 LTS, but should work in other/future Ubuntu releases as well.

Let's install the system packages needed by running the following command on the Terminal/console:

sudo apt-get update

sudo apt-get install -y build-essential cmake python-dev python-numpy python-opengl libboost-all-dev zlib1g-dev libsdl2-dev libav-tools xorg-dev libjpeg-dev swig

This command will install the prerequisite system packages. Note that the -y flag will automatically say yes to confirm the installation of the package, without asking you to confirm manually. If you want to review the packages that are going to be installed for some reason, you may run the command without the flag.

Instructions for macOS

On macOS, the number of additional system packages that need to be installed is less than with Ubuntu systems.

Run the following commands from a Terminal:

brew install cmake boost sdl2 swig wget

brew install boost-python --with-python3

These commands will install the prerequisite system packages.

The robotics and control environment in OpenAI Gym make use of Multi-Joint dynamics with Contact (MuJoCo) as the physics engine to simulate the rigid body dynamics and other features. We briefly had a look at MuJoCo environments in Chapter 1, Introduction to Intelligent Agents and Learning Environments and learned that you can develop algorithms that can make a 2D robot walk, run, swim, or hop, or a 3D multi-legged robot walk or run using the MuJoCo environment. MuJoCo is a proprietary engine and therefore needs a license. Fortunately, we can get a free 30-day license!
Also, if you are a student, they offer a 1 year free MuJoCo Pro personal license, which is even better! For others, after the 30 days, sadly it costs a hefty sum (~$500 USD) for a 1 -year license. We will not be using the MuJoCo environments in this book because not everyone may be able to get hold of a license. You can always apply what you learn in this book regarding other environments to the MuJoCo environments if you have a license. If you plan to use these environments, you will have to follow the instructions in the MuJoCo installation section next. If not, you can skip it and go to the next section to set up OpenAI Gym.

MuJoCo installation

I hope you read the previous information box. MuJoCo is one odd library that we will encounter in this book because, unlike other libraries and software we use as part of this book, MuJoCo requires a license to use. The Python interface for MuJoCo, available in the Gym library, is compatible only with MuJoCo version 1.31 as of the time this chapter was written, even though the latest available MuJoCo version is higher (1.50 at the time of writing this chapter). Follow these two steps to set up MuJoCo for use with OpenAI Gym environments:

1. Download MuJoCo 1.31 for your platform (Linux/macOS) from this URL: https://www.roboti.us/index.html
2. Obtain a MuJoCo Pro license from this URL: https://www.roboti.us/license.html

Completing the OpenAI Gym setup

Let's update our version of pip first:

(rl_gym_book) praveen@ubuntu:~$ pip install --ignore-installed pip

Then, let's download the source code of OpenAI Gym from the GitHub repository into our home folder:

(rl_gym_book) praveen@ubuntu:~$cd ~

(rl_gym_book) praveen@ubuntu:~$git clone https://github.com/openai/gym.git

(rl_gym_book) praveen@ubuntu:~$cd gym

If you get an error saying git command not found or something similar, you might have to install Git. On Ubuntu systems, you can install it by running this command, sudo apt-get install git. On macOS, if you don't have Git installed already, it will prompt you to install it when you run the git clone command.

We are now in the final stage of a complete Gym installation! If you got a MuJoCo license and followed the MuJoCo installation instructions successfully, then you can go ahead and complete a full installation by running the following command:

(rl_gym_book) praveen@ubuntu:~/gym$pip install -e '.[all]'

If you did not install MuJoCo, then this command will return errors. We will be installing the Gym environments that we will be using, other than MuJoCo (which requires a license to use). Make sure that you are still in the gym directory under your home folder, and also make sure that you are still inside the rl_gym_book conda environment. Your prompt should include the rl_gym_book prefix as follows, where ~/gym means that the prompt is at the gym directory under the home folder:

(rl_gym_book) praveen@ubuntu:~/gym$

Here is a table summarizing the installation commands for installing the environments that have been discussed in Chapter 1, Introduction to Intelligent Agents and Learning Environments.

Environment	Installation command
Atari	pip install -e '.[atari]'
Box2D	pip install -e '.[box2d]' conda install -c https://conda.anaconda.org/kne pybox2d
Classic control	pip install -e '.[classic_control]'
MuJoCo (requires license)	pip install -e '.[mujoco]'
Robotics (requires license)	pip install -e '.[robotics]'

Let's go ahead and install the environments we do not need a license to use. Run the following commands to install Atarti, Box2D, and the classic control environments:

(rl_gym_book) praveen@ubuntu:~/gym$pip install -e '.[atari]'

(rl_gym_book) praveen@ubuntu:~/gym$pip install -e '.[box2d]'

(rl_gym_book) praveen@ubuntu:~/gym$conda install -c https://conda.anaconda.org/kne pybox2d

(rl_gym_book) praveen@ubuntu:~/gym$pip install -e '.[classic_control]'

We are all set! We'll quickly run a check to make sure the installation went fine. Copy and paste the following code snippet into a file named test_box2d.py under the ~/rl_gym_book directory:

#!/usr/bin/env python
import gym
env = gym.make('BipedalWalker-v2')
env.reset()
for _ in range(1000):
    env.render()
    env.step(env.action_space.sample())

Run that code using the following commands:

(rl_gym_book) praveen@ubuntu:~/gym$cd ~/rl_gym_book

(rl_gym_book) praveen@ubuntu:~/rl_gym_book$python test_box2d.py

You will see a window pop up that shows the BipedalWalker-v2 environment and the walker trying to randomly perform some actions:

So, we have the Gym environment set up. What's next, you may ask. In the next section, we will set up the tools and libraries we need to develop deep reinforcement learning agents to train in these environments!