Markov decision processes
This first section also describes the basic concepts you need to know in order to understand, develop, and apply the hidden Markov model. The foundation of the Markovian universe is the concept known as the Markov property.
The Markov property
The Markov property is a characteristic of a stochastic process where the conditional probability distribution of a future state depends on the current state and not on its past states. In this case, the transition between the states occurs at a discrete time, and the Markov property is known as the discrete Markov chain.
The first order discrete Markov chain
The following example is taken from Introduction to Machine Learning, E. Alpaydin [7:2].
Let's consider the following use case. N balls of different colors are hidden in N boxes (one each). The balls can have only three colors (Blue, Red, and Green). The experimenter draws the balls one by one. The state of the discovery process is defined by the color of the latest ball drawn...
