Chapter 5, Picking up the Toys

1. The origin of the Q-learning title is the doctoral thesis of Christopher John Cornish Hellaby Watkins from King’s College, London in May, 1989. Evidently, the Q just stands for “quantity”.
2. Only pick the Q-states that are relevant and follow-ons to the current state. If one of the states is impossible to reach from the current position, or state, then don’t consider it.
3. If the learning rate is too small, the training can take a very long time. If the learning rate is too large, the system does not learn a path, but instead “jumps around” and may miss the minimum or optimum solution. If the learning rate is too big, the solution may not converge, or suddenly drop off.

4. The discount factor works by decreases the reward as the path length gets longer. It is usually a value just short of 1.0 (for example, 0...