Training deep neural networks
Historically, the scientific community has understood that deeper networks have greater representational power compared to shallow ones. However, there were various challenges in training networks with more than a few hidden layers. We now know that we can successfully train DNNs using a combination of gradient descent and backpropagation, just as we discussed in Chapter 2. In this section, we’ll see how to improve them so that we can solve some of the problems that exist uniquely for DNNs and not shallow NNs.
The first edition of this book included networks such as Restricted Boltzmann Machines (RBMs) and Deep Belief Networks (DBNs). They were popularized by Geoffrey Hinton, a Canadian scientist, and one of the most prominent DL researchers. Back in 1986, he was also one of the inventors of backpropagation. RBMs are a special type of generative NN, where the units are organized into two layers, namely visible and hidden. Unlike feedforward networks...
