3

Deep Learning Fundamentals

In this chapter, we will introduce deep learning (DL) and deep neural networks (DNNs) – that is, neural networks (NNs) with multiple hidden layers. You might be wondering what the point of using more than one hidden layer is, given the universal approximation theorem. This is in no way a naive question, and for a long time, NNs were used in that way.

Without going into too much detail, one reason is that approximating a complex function might require a huge number of units in the hidden layer, making it impractical to use. There is also another, more important, reason for using deep networks, which is not directly related to the number of hidden layers, but to the level of learning. A deep network does not simply learn to predict output Y given input, X; it also understands the basic features of the input. It’s able to learn abstractions of features of input samples, understand the basic characteristics of the samples, and make predictions...