Up until now, we have been exploring the finite Markov Decision Process or finite MDP. These types of problems are all well and good for simulation and toy problems, but they don't show us how to tackle real-world problems. Real-world problems can be broken down or discretized into finite MDPs, but real problems are not finite. Real problems are infinite, that is, they define no discrete simple states such as showering or having breakfast. Infinite MDPs model problems in what we call continuous space or continuous action space, that is, in problems where we think of a state as a single point in time and state defined as a slice of that time. Hence, the discrete task of eat breakfast could be broken down to each time step including individual chewing actions.
Solving an infinite MDP or continuous space problem is not trivial with our current...
