We will solve the MountainCar problem using PPO. MountainCar involves a car trapped in the valley of a mountain. It has to apply throttle to accelerate against gravity and try to drive out of the valley up steep mountain walls to reach a desired flag point on the top of the mountain. You can see a schematic of the MountainCar problem from OpenAI Gym at https://gym.openai.com/envs/MountainCar-v0/.
This problem is very challenging, as the agent cannot just apply full throttle from the base of the mountain and try to reach the flag point, as the mountain walls are steep and gravity will not allow the car to achieve sufficient enough momentum. The optimal solution is for the car to initially go backward and then step on the throttle to pick up enough momentum to overcome gravity and successfully drive out of the valley. We will see that the...