Off-Policy MC Control

The algorithm for the off-policy MC control method is given as follows:

1. Initialize the Q function Q(s, a) with random values and set the behavior policy b to be epsilon-greedy, set the target policy to be greedy policy and initialize the cumulative weights as C(s, a) = 0
2. For M number of episodes:
   1. Generate an episode using the behavior policy b
   2. Initialize return R to 0 and weight W to 1
   3. For each step t in the episode, t = T – 1, T – 2, . . . , 0:
      1. Compute the return as R = R + r_t+1
      2. Update the cumulative weights to C(s_t, a_t) = C(s_t, a_t) +W
      3. Update the Q value to
      4. Compute the target policy
      5. If then break
      6. Update the weight to
3. Return the target policy