Introducing Gaussian processes

The Gaussian process (GP) can be thought of as an alternative Bayesian approach to regression problems. They are also referred to as infinite dimensional Gaussian distributions. GP defines a priori over functions that can be converted into a posteriori once we have observed a few data points. Although it doesn’t seem possible to define distributions over functions, it turns out that we only need to define distributions over a function's values at observed data points.

Formally, let's say that we observed a function,

, at n values

 as

. The function is a GP if all of the values, 

, are jointly Gaussian, with a mean of 

 and a covariance of 

  given by

. Here, the 

 function defines how two variables are related to each other. We will discuss different kinds of kernels later in this section. The joint Gaussian distribution of many Gaussian variables is also known as Multivariate Gaussian. 

From the previous temperature example, we can imagine that various functions...