Generative models
A generative model in neural processing is a model that generates data given a noise vector z as input:
The purpose of the training is to find the parameters to generate data as close as possible to the real data.
Applications of generative networks include data dimensionality reduction, synthetic data generation, unsupervised feature learning, and pre-training / training efficiency. Pre-training helps generalization because pre-training focuses on the patterns in the data, and less on the data-label relation.
Restricted Boltzmann Machines
A Restricted Boltzmann Machine is the simplest generative net, composed of one fully connected hidden layer, as shown in the picture:
The full Boltzmann Machines have also hidden-to-hidden and visible-to-visible loop connections, while the Restricted version does not have any.
In the general case, RBM are defined as energy-based models, which means that they define a probability distribution through an energy function:
Z is the partition...