Probabilistic programming

Bayesian statistics is conceptually very simple: we have some data that is fixed, in the sense that we cannot change what we have measured, and we have parameters whose values are of interest to us and hence we explore their plausible values. All the uncertainties we have are modeled using probabilities. In other statistical paradigms, there are different types of unknown quantities; in the Bayesian framework everything that is unknown is treated the same. If we do not know a quantity we assign a probability distribution to it. Then, Bayes' theorem is used to transform the prior probability distribution (what we know about a given problem before observing the data), into a posterior distribution (what we know after observing data). In other words, Bayesian statistics is a form of learning.

Although conceptually simple, fully probabilistic models often lead to analytically intractable expressions. For many years, this was a real problem and was probably one...