Neural networks have been around for many years, and they've gone through several periods during which they've fallen in and out of favor. But recently, they have steadily gained ground over many other competing machine learning algorithms. This resurgence is due to having computers that are fast, the use of graphical processing units (GPUs) versus the most traditional use of computing processing units (CPUs), better algorithms and neural net design, and increasingly larger datasets that we'll see in this book. To get an idea of their success, let's take the ImageNet Large-Scale Visual Recognition Challenge (http://image-net.org/challenges/LSVRC/, or just ImageNet). The participants train their algorithms using the ImageNet database. It contains more than one million high-resolution color images in over a thousand categories (one...

We can describe a neural network as a mathematical model for information processing. As discussed in Chapter 1, Machine Learning – an Introduction, this is a good way to describe any ML algorithm, but, in this chapter, well give it a specific meaning in the context of neural networks. A neural net is not a fixed program, but rather a model, a system that processes information, or inputs. The characteristics of a neural network are as follows:

• Information processing occurs in its simplest form, over simple elements called neurons.
• Neurons are connected and they exchange signals between them through connection links.
• Connection links between neurons can be stronger or weaker, and this determines how information is processed.
• Each neuron has an internal state that is determined by all the incoming connections from other neurons.
• Each neuron...

We have seen how neural networks can map inputs onto determined outputs, depending on fixed weights. Once the architecture of the neural network has been defined and includes the feed forward network, the number of hidden layers, the number of neurons per layer, and the activation function, we'll need to set the weights, which, in turn, will define the internal states for each neuron in the network. First, we'll see how to do that for a 1-layer network using an optimization algorithm called gradient descent, and then we'll extend it to a deep feed forward network with the help of backpropagation.

The general concept we need to understand is the following:

Every neural network is an approximation of a function, so each neural network will not be equal to the desired function, but instead will differ by some value called error. During training...

In this chapter, we introduced neural networks in detail and we mentioned their success vis-à-vis other competing algorithms. Neural networks are comprised of interconnected neurons (or units), where the weights of the connections characterize the strength of the communication between different neurons. We discussed different network architectures, and how a neural network can have many layers, and why inner (hidden) layers are important. We explained how the information flows from the input to the output by passing from each layer to the next based on the weights and the activation function, and finally, we showed how to train neural networks, that is, how to adjust their weights using gradient descent and backpropagation.

In the next chapter, we'll continue discussing deep neural networks, and we'll explain in particular the meaning of deep in deep learning...