Off-Policy TD Control – Q Learning

The algorithm for off-policy TD control – Q learning is given as follows:

1. Initialize a Q function Q(s, a) with random values
2. For each episode:
   1. Initialize the state s
   2. For each step in the episode:
      1. Extract a policy from Q(s, a) and select an action a to perform in the state s
      2. Perform the action a, move to the new state , and observe the reward r
      3. Update the Q value to
      4. Update (update the next state to the current state s)
      5. If s is not a terminal state, repeat steps 1 to 5