Variational inference

Variational inference is a statistical technique that’s used to approximate difficult-to-compute probability distributions with simpler, more tractable distributions. A typical, simple function to approximate a complex distribution is the Jensen’s inequality.

In mathematics, you can use Jensen’s inequality to get a lower bound for a convexity-based function. Figure 10.6 explains this property:

Figure 10.6 – Jensen’s inequality

For any convex function, f(x), the expected value (average) of f(x) for a random variable, x, is greater than or equal to the result of the function applied to the average of those numbers. See the two points, x 1 and x 2. Any point between x 1 and x 2 is t x 1 + (1 − t) x 2, where the weight is t ∈ (0,1). The projected value for t x 1 + (1 − t) x 2 is f(t x 1 + (1 − t) x 2). With the weight, t...