PPO-Penalty

The algorithm for the PPO-penalty method is given as follows:

1. Initialize the policy network parameter and value network parameter and initialize the penalty coefficient and the target KL divergence
2. For iterations :
   1. Collect some N number of trajectories following the policy
   2. Compute the return (reward-to-go) R_t
   3. Compute
   4. Compute the gradient of the objective function
   5. Update the policy network parameter using gradient ascent,
   6. If d is greater than or equal to , then we set ; if d is lesser than or equal to , then we set,
   7. Compute the mean squared error of the value network:
   8. Compute the gradients of the value network
   9. Update the value network parameter using gradient descent,