Simulating Atari environments

To get started with Gym, let's play some Atari games with it.

The Atari environments (https://gym.openai.com/envs/#atari) are a variety of Atari 2600 video games, such as Alien, AirRaid, Pong, and Space Race. If you have ever played Atari games, this recipe should be fun for you, as you will play an Atari game, Space Invaders. However, an agent will act on your behalf.

How to do it...

Let's simulate the Atari environments by following these steps:

1. To run any atari environment for the first time, we need to install the atari dependencies by running this command in the Terminal:

pip install gym[atari]

Alternatively, if you used the second approach in the previous recipe to install gym, you can run the following command instead:

pip install -e '.[atari]'

2. After installing the Atari dependencies, we import the gym library in Python:

>>> import gym

3. Create an instance of the SpaceInvaders environment:

>>> env = gym.make('SpaceInvaders-v0')

4. Reset the environment:

>>> env.reset()
 array([[[ 0,  0, 0],
         [ 0, 0,  0],
         [ 0, 0,  0],
         ...,
         ...,
         [80, 89, 22],
         [80, 89, 22],
         [80, 89, 22]]], dtype=uint8)

As you can see, this also returns the initial state of the environment.

5. Render the environment:

>>> env.render()
True

You will see a small window popping up, as follows:

As you can see from the game window, the spaceship starts with three lives (the red spaceships).

6. Randomly pick one possible move and execute the action:

>>> action = env.action_space.sample()
>>> new_state, reward, is_done, info = env.step(action)

The step() method returns what happens after an action is taken, including the following:

• New state: The new observation.
• Reward: The reward associated with that action in that state.
• Is done: A flag indicating whether the game ends. In a SpaceInvaders environment, this will be True if the spaceship has no more lives left or all the aliens are gone; otherwise, it will remain False.
• Info: Extra information pertaining to the environment. This is about the number of lives left in this case. This is useful for debugging.

Let’s take a look at the is_done flag and info:

>>> print(is_done)
False
>>> print(info)
{'ale.lives': 3}

Now we render the environment:

>>> env.render()
 True

The game window becomes the following:

You won't notice much difference in the game window, because the spaceship just made a move.

7. Now, let's make a while loop and let the agent perform as many actions as it can:

>>> is_done = False
>>> while not is_done:
...     action = env.action_space.sample()
...     new_state, reward, is_done, info = env.step(action)
...     print(info)
...     env.render()
{'ale.lives': 3}
True
{'ale.lives': 3}
True
……
……
{'ale.lives': 2}
True
{'ale.lives': 2}
True
……
……
{'ale.lives': 1}
True
{'ale.lives': 1}
True

Meanwhile, you will see that the game is running, and the spaceship keeps moving and shooting, and so do the aliens. And it is pretty fun to watch, too. At the end, when the game ends, the window looks like the following:

As you can see, we scored 150 points in this game. You may get a higher or lower score than this because the actions the agent performs are all randomly selected.

We also confirm that no lives are left with the last piece of info:

>>> print(info)
{'ale.lives': 0}

How it works...

Using Gym, we can easily create an environment instance by calling the make() method with the name of the environment as the parameter.

As you may have noticed, the actions that the agent performs are randomly chosen using the sample() method.

Note that, normally, we would have a more sophisticated agent guided by reinforcement learning algorithms. Here, we just demonstrated how to simulate an environment, and how an agent takes actions regardless of the outcome.

Run this a few times and see what we get:

>>> env.action_space.sample()
0
>>> env.action_space.sample()
3
>>> env.action_space.sample()
0
>>> env.action_space.sample()
4
>>> env.action_space.sample()
2
>>> env.action_space.sample()
1
>>> env.action_space.sample()
4
>>> env.action_space.sample()
5
>>> env.action_space.sample()
1
>>> env.action_space.sample()
0

There are six possible actions in total. We can also see this by running the following command:

>>> env.action_space
Discrete(6)

Actions from 0 to 5 stand for No Operation, Fire, Up, Right, Left, and Down, respectively, which are all the moves the spaceship in the game can do.

The step() method will let the agent take the action that is specified as its parameter. The render() method will update the display window based on the latest observation of the environment.

The observation of the environment, new_state, is represented by a 210 x 160 x 3 matrix, as follows:

>>> print(new_state.shape)
(210, 160, 3)

This means that each frame of the display screen is an RGB image of size 210 x 160.

There's more...

You may wonder why we need to install Atari dependencies. In fact, there are a few more environments that do not accompany the installation of gym, such as Box2d, Classic control, MuJoCo, and Robotics.

Take the Box2d environments, for example; we need to install the Box2d dependencies before we first run the environments. Again, two installation approaches are as follows:

pip install gym[box2d]
pip install -e '.[box2d]'

After that, we can play around with the LunarLander environment, as follows:

>>> env = gym.make('LunarLander-v2')
>>> env.reset()
array([-5.0468446e-04,  1.4135642e+00, -5.1140346e-02,  1.1751971e-01,
       5.9164839e-04,  1.1584054e-02, 0.0000000e+00,  0.0000000e+00],
     dtype=float32)
>>> env.render()

A game window will pop up:

See also

If you are looking to simulate an environment but are not sure of the name you should use in the make() method, you can find it in the table of environments at https://github.com/openai/gym/wiki/Table-of-environments. Besides the name used to call an environment, the table also shows the size of the observation matrix and the number of possible actions. Have fun playing around with the environments.