Demystifying MDPs

The technical purpose of Q-learning is to discover solutions for a type of optimization problem called an MDP.

When we talk about states and the actions that we can take from states, we are discussing concepts developed in the context of MDPs (and the Markov chains and other state space models that they are derived from).

Control processes

An MDP is itself a type of problem called a control process. Broadly speaking, a control process is designed to optimize a value or a set of values within a set of limitations. The following diagram shows a model for a generalized control process:

A real-world example of such an optimization process might be maximizing a company's profit by knowing its gross sales...