Deep Q learning from demonstrations

The algorithm for Deep Q Learning from Demonstrations (DQfD) is given as follows:

1. Initialize the main network parameter
2. Initialize the target network parameter by copying the main network parameter
3. Initialize the replay buffer with the expert demonstrations
4. Set d, the number of time steps we want to delay updating the target network parameter
5. Pre-training phase: For steps t = 1, 2, . . ., T:
   1. Sample a minibatch of experience from the replay buffer
   2. Compute the loss J(Q)
   3. Update the parameter of the network using gradient descent
   4. If t mod d = 0:

      Update the target network parameter by copying the main network parameter

6. Training phase: For steps t =1, 2, . . ., T:
   1. Select an action
   2. Perform the selected action and move to the next state, observe the reward, and store this transition information in the replay buffer
   ...