Introducing autoencoders

An autoencoder neural network is an unsupervised learning algorithm that sets the target values to be equal to the input values. Hence, the autoencoder attempts to an approximation of an identity function.

Learning an identity function does not seem to be a worthwhile exercise; however, by placing constraints on the network, such as limiting the number of hidden units, we can discover interesting structures about the data. The key components of an autoencoder are depicted in this figure:

The original input, the compressed representation, and the output layers for an autoencoder are also illustrated in the following figure. More specifically, this figure represents a situation where, for example, an input image has pixel-intensity values from a 10×10 image (100 pixels), and there are 50 hidden units in layer two. Here, the network is forced to learn a "compressed" representation of the input, in which it must attempt to "reconstruct" the 100-pixel input using 50 hidden...