Deep learning is just a modern name for artificial neural networks with many layers. What is deep in deep learning though? It is basically due to the structure of the artificial neural network (ANN). ANN consists of some n number of layers to perform any computation. We can build an ANN with several layers where each layer is responsible for learning the intricate patterns in the data. Due to the computational advancements, we can build a network even with 100s or 1000s of layers deep. Since the ANN uses deep layers to perform learning we call it as deep learning and when ANN uses deep layers to learn we call it as a deep network. We have learned that deep learning is a subset of machine learning. How does deep learning differ from machine learning? What makes deep learning so special and popular?

The success of machine learning lies in the right set of...

Before going ahead, first, we will explore what are neurons and how neurons in our brain actually work, and then we will learn about artificial neurons.

A neuron can be defined as the basic computational unit of the human brain. Neurons are the fundamental units of our brain and nervous system. Our brain encompasses approximately 100 billion neurons. Each and every neuron is connected to one another through a structure called a synapse, which is accountable for receiving input from the external environment, sensory organs for sending motor instructions to our muscles, and for performing other activities.

A neuron can also receive inputs from the other neurons through a branchlike structure called a dendrite. These inputs are strengthened or weakened; that is, they are weighted according to their importance and then they are summed together in...

While neurons are really cool, we cannot just use a single neuron to perform complex tasks. This is the reason our brain has billions of neurons, stacked in layers, forming a network. Similarly, artificial neurons are arranged in layers. Each and every layer will be connected in such a way that information is passed from one layer to another.

A typical ANN consists of the following layers:

• Input layer
• Hidden layer
• Output layer

Each layer has a collection of neurons, and the neurons in one layer interact with all the neurons in the other layers. However, neurons in the same layer will not interact with one another. This is simply because neurons from the adjacent layers have connections or edges between them; however, neurons in the same layer do not have any connections. We use the term nodes or units to represent the neurons in the artificial neural network...

An activation function, also known as a transfer function, plays a vital role in neural networks. It is used to introduce non-linearity in neural networks. As we learned before, we apply the activation function to the input, which is multiplied by weights and added to the bias, that is, , where z = (input * weights) + bias and is the activation function. If we do not apply the activation function, then a neuron simply resembles the linear regression. The aim of the activation function is to introduce a non-linear transformation to learn the complex underlying patterns in the data.

Now let's look at some of the interesting commonly used activation functions.

The sigmoid...

In this section, we will see how an ANN learns where neurons are stacked up in layers. The number of layers in a network is equal to the number of hidden layers plus the number of output layers. We don't take the input layer into account when calculating the number of layers in a network. Consider a two-layer neural network with one input layer, , one hidden layer, , and one output layer, , as shown in the following diagram:

Let's consider we have two inputs, and , and we have to predict the output, . Since we have two inputs, the number of neurons in the input layer will be two. We set the number of neurons in the hidden layer to four, and, the number of neurons in the output layer to one. Now, the inputs will be multiplied by weights, and then we add bias and propagate the resultant value to the hidden layer where the activation function...

If the cost or loss is very high, then it means that our network is not predicting the correct output. So, our objective is to minimize the cost function so that our neural network predictions will be better. How can we minimize the cost function? That is, how can we minimize the loss/cost? We learned that the neural network makes predictions using forward propagation. So, if we can change some values in the forward propagation, we can predict the correct output and minimize the loss. But what values can we change in the forward propagation? Obviously, we can't change input and output. We are now left with weights and bias values. Remember that we just initialized weight matrices randomly. Since the weights are random, they are not going to be perfect. Now, we will update these weight matrices ( and ) in such a way that our neural network gives a correct...

We just learned how gradient descent works and how to code the gradient descent algorithm from scratch for a simple two-layer network. But implementing gradient descent for complex neural networks is not a simple task. Apart from implementing, debugging a gradient descent for complex neural network architecture is again a tedious task. Surprisingly, even with some buggy gradient descent implementations, the network will learn something. However, apparently, it will not perform well compared to the bug-free implementation of gradient descent.

If the model does not give us any errors and learns something even with buggy implementations of the gradient descent algorithm, how can we evaluate and ensure that our implementation is correct? That is why we use the gradient checking algorithm. It will help us to validate our implementation...

Putting all the concepts we have learned so far together, we will see how to build a neural network from scratch. We will understand how the neural network learns to perform the XOR gate operation. The XOR gate returns 1 only when exactly only one of its inputs is 1, else it returns 0 as shown in the following table:

To perform the XOR gate operation, we build a simple two-layer neural network, as shown in the following diagram. As you can see, we have an input layer with two nodes: a hidden layer with five nodes and an output layer comprising one node:

We will understand step-by-step how a neural network learns the XOR logic:

1. First, import the libraries:

import...

We started off the chapter by understanding what deep learning is and how it differs from machine learning. Later, we learned how biological and artificial neurons work, and then we explored what is input, hidden, and output layer in the ANN, and also several types of activation functions.

Going ahead, we learned what forward propagation is and how ANN uses forward propagation to predict the output. After this, we learned how ANN uses backpropagation for learning and optimizing. We learned an optimization algorithm called gradient descent that helps the neural network to minimize the loss and make correct predictions. We also learned about gradient checking, a technique that is used to evaluate the gradient descent. At the end of the chapter, we implemented a neural network from scratch to perform the XOR gate operation.

In the next chapter, we will learn about one of...

Let's evaluate our newly acquired knowledge by answering the following questions:

1. How does deep learning differ from machine learning?
2. What does the word deep mean in deep learning?
3. Why do we use the activation function?
4. Explain dying ReLU problem.
5. Define forward propagation.
6. What is back propagation?
7. Explain gradient checking.

You can also check out some of these resources for more information:

• Understand more about gradient descent from this amazing video: https://www.youtube.com/watch?v=IHZwWFHWa-w
• Learn about implementing a neural network from scratch to recognize handwritten digits: https://github.com/sar-gupta/neural-network-from-scratch