On-Policy MC Control – Epsilon-Greedy

The algorithm for on-policy MC control with the epsilon-greedy policy is given as follows:

1. Let total_return(s, a) be the sum of the return of a state-action pair across several episodes and N(s, a) be the number of times a state-action pair is visited across several episodes. Initialize total_return(s, a) and N(s, a) for all state-action pairs to zero and initialize a random policy .
2. For M number of iterations:
   1. Generate an episode using policy
   2. Store all the rewards obtained in the episode in the list called rewards
   3. For each step t in the episode:

      If (s_t, a_t) is occurring for the first time in the episode:

      1. Compute the return of a state-action pair, R(s_t, a_t) = sum(rewards[t:]).
      2. Update the total return of the state-action pair as total_return(s_t, a_t) = total_return(s_t, a_t) + R(s_t, a_t).
      3. Update the counter as N(s_t, a_t) = N(s_t, a_t) + 1.
      ...