Soft Actor-Critic

The algorithm for Soft Actor-Critic (SAC) is given as follows:

1. Initialize the main value network parameter , the Q network parameters and , and the actor network parameter
2. Initialize the target value network by just copying the main value network parameter
3. Initialize the replay buffer
4. For N number of episodes, repeat step 5
5. For each step in the episode, that is, for t = 0, . . ., T – 1
   1. Select action a based on the policy , that is,
   2. Perform the selected action a, move to the next state , get the reward r, and store the transition information in the replay buffer
   3. Randomly sample a minibatch of K transitions from the replay buffer
   4. Compute target state value
   5. Compute the loss of value network and update the parameter using gradient descent,
   6. Compute the target Q value
   7. Compute the loss of the Q networks and update the parameter using gradient descent...