Adding in experience replay

Experience replay takes an agent's state-action observations and stores them in a list or table. It then goes through the table periodically and uses some of these observations, chosen at random, to update the Q-table.

Recall that right now our model is updating the Q-table after every action step. We're going to add an experience replay method to our update function that will update our Q-values in randomly selected batches instead.

About experience replay

Why are we choosing to use experience replay? What advantage does it provide over updating the Q-values at every iteration?

Let's first take a look at the loss function for the deep Q-network:

Essentially, instead of running...