Network size and representational power

In the previous diagram, we can see four types of basic autoencoding architectures. Shallow autoencoders (an extension of shallow neural networks) are defined by having just one hidden layer of neurons, whereas deep autoencoders can have many layers that perform the encoding and decoding operations. Recall from the previous chapters that deeper neural networks may benefit from additional representational power compared to their shallow counterparts. Since autoencoders qualify as a specific breed of feed-forward networks, this also holds true for them. Additionally, it has been noted that deeper autoencoders may exponentially reduce the computational resources that are required for the network to learn to represent its inputs. It may also greatly reduce the number of training samples that are required for the network to learn a rich compressed...