Overview of Markov processes

Markov's decision-making process is defined as a discrete-time stochastic control process. In Chapter 2, Understanding Randomness and Random Numbers, we said that stochastic processes are numerical models used to simulate the evolution of a system according to random laws. Natural phenomena, both by their very nature and by observation errors, are characterized by random factors. These factors introduce a random number into the observation of the system. This random factor determines an uncertainty in the observation since it is not possible to predict with certainty what the result will be. In this case, we can only say that it will assume one of the many possible values with a certain probability.

If starting from an instant t in which an observation of the system is made, the evolution of the process will depend only on t, while it will not be influenced by the previous instants. Here, we can say that the stochastic process is Markovian.

Important...