In this recipe, we will go through a full example of coding an MCMC algorithm ourselves. This will give us a much better grasp of MCMC mechanics.
In the Bayesian world, we put prior densities and use data to augment those priors, and get posterior densities. The problem is that there are only a few occasions where we can calculate those posterior densities analytically—these are called conjugate families.
The Bayesian problem can be formulated as recovering the conditional density of the parameter given data. This is equal to the ratio of the joint density of the parameters and the data divided by the marginal density of the data. This follows from Bayes theorem, which states that we can invert a conditional probability by dividing the joint probability by the appropriate marginal density. This is the density that we want to compute, but even if...
