Markov decision processes

This first section also describes the basic concepts you need to know to understand, develop, and apply the hidden Markov model, starting with the Markov property.

The Markov property

The Markov property is a characteristic of a stochastic process where the conditional probability distribution of a future state depends on the current state and not on its past states. In this case, the transition between the states occurs at a discrete time, and the Markov property is known as the discrete Markov chain.

The first-order discrete Markov chain

The following example is taken from Introduction to Machine Learning by E. Alpaydin [7:3].

Let's consider the following use case. N balls of different colors are hidden in N boxes (one each). The balls can have only three colors {Blue, Red, and Green}. The experimenter draws the balls one by one. The state of the discovery process is defined by the color of the latest ball drawn from one of the boxes: S₀ = Blue, S₁ = Red, and S₂...