Reinforcement Learning is rooted in animal and behavioral psychology, where it is used in many applications of Machine Learning, from games and simulations to control optimization, information theory, statistics, and many more areas every day. RL, at its most basic level, describes an agent acting with an environment that receives either positive or negative rewards based on those actions. The following is a diagram showing the stateless RL model:

Stateless Reinforcement Learning

Conveniently, our multi-armed bandit problem we built in the last chapter fits well with this simpler form of RL. That problem only had a single state, or what we refer to as a one-step RL problem. Since the agent doesn't need to worry about state, we can greatly simplify our RL equations to just write the value of each action using the following equation:

Consider the following...

Our next step in understanding RL will be for us to look at the contextual bandit problem. A contextual bandit is the multi-armed bandit problem, with multiple bandits each producing different rewards. This type of problem has many applications in online advertising, where each user is thought of as a different bandit, with the goal being to present the best advertisement for that user. To model the context of the bandit, and which bandit it is, we add the concept of state. Where we now interpret state to represent each of our different bandits. The following diagram shows the addition of state in the Contextual Bandit problem and where it lies on our path to glory:

Stateless, Contextual and Full RL models

You can see in the preceding diagram that we now need to determine the state before evaluating an action. If you recall from earlier, the Value...

One of the dilemmas we face in RL is the balance between exploring all possible actions and exploiting the best possible action. In the multi-armed bandit problem, our search space was small enough to do this with brute force, essentially just by pulling each arm one by one. However, in more complex problems, the number of states could exceed the number of atoms in the known universe. Yes, you read that correctly. In those cases, we need to establish a policy or method whereby we can balance the exploration and exploitation dilemma. There are a few ways in which we can do this, and the following are the most common ways you can approach this:

• Greedy Optimistic: The agent initially starts with high values in its q table. This forces the agent to explore all states at least once, since the agent otherwise always greedily chooses the best action.
• Greedy...

If you have studied Reinforcement Learning previously, you may have already come across the term MDP, for the Markov Decision Process, and the Bellman equation. An MDP is defined as a discrete time stochastic control (https://en.wikipedia.org/wiki/Stochastic) process, but, put more simply, it is any process that makes decisions based on some amount of uncertainty combined with mathematics. This rather vague description still fits well with how we have been using RL to make decisions. In fact, we have been developing MDP processes all of this chapter, and you should be fairly comfortable with the concept now.

Up until now, we have only modeled the partial RL or one-step problem. Our observation of state was only for one step or action, which meant that the agent always received an immediate reward or punishment. In a full RL problem, an agent may need...

Typically, Q-Learning is taught using a grid problem such as the one we looked at it in the previous section. Here, though, we want something a little more complex and abstract that also allows you, the reader, to build on it and explore it further. We have put together an interesting example where we represent our bandits as rooms or objects with a number of exit options. This example could also very easily represent a dungeon or another connected room structure that you need to navigate an agent through. Follow these steps to get started on building the connected agents exercise:

1. From the menu, select Assets -> Import Package -> Custom Package..., then navigate to the book's downloaded source code and import the Chapter_2_Connected_Bandits_unitypackage. This is the example, which has been fully constructed already for you. Apologies...

Be sure to complete the following questions or exercises on your own:

1. Extend the bandit cube maze in the last section with your own design. Make sure to keep all the cubes connected so that the agent has a clear path to the end.
2. Think of another problem in gaming, simulation, or otherwise, where you could use RL and the Q-Learning algorithm to help an agent learn to solve this problem. This is just a thought exercise, but give yourself a huge pat on the back if you build a demo.
3. Add new properties for the Exploration Epsilon minimum and the amount of change per decision step. Remember, these are the parameters we hard-coded in order to decrease the epsilon-greedy exploration value.
4. Add the ability to show the Q values on the individual BanditCube objects. If...

In this chapter, we dove into working with the ML-Agents framework by developing a heuristic brain that uses an ML technique called Reinforcement Learning to solve various fundamental learning problems. We first explored the classic multi-armed bandit problem in order to first introduce the concepts of RL. Then, we expanded on this problem by adding context or the sense of state. This required us to modify our Value function by adding state and turning our function into a Q function. While this algorithm worked well to solve our simple learning problem, it was not sufficient for more complex problems with delayed rewards. Introducing delayed rewards required us to look at the Bellman equation and understand how we could discount rewards over an agent's steps, thus providing the agent with Q value breadcrumbs as a way for it to find its way home. Finally, we loaded...