To understand how to train a vanilla neural network, we will go through the task of predicting the label of a digit in the MNIST dataset, which is a popular dataset of images of digits (one digit per image) and the corresponding label of the digit that is contained in the image.

Training a neural network is done in the following steps:

1. Import the relevant packages and datasets
2. Preprocess the targets (convert them into one-hot encoded vectors) so that we can perform optimization on top of them:
   • We shall be minimizing categorical cross entropy loss
3. Create train and test datasets:
   • We have the train dataset so that we create a model based on it
   • The test dataset is not seen...

Scaling a dataset is a process where we limit the variables within a dataset to ensure they do not have a very wide range of different values. One way to achieve this is to divide each variable in the dataset by the maximum value of the variable. Typically, neural networks perform well when we scale the input datasets.

In this section, let's understand the reason neural networks perform better when the dataset is scaled.

To understand the impact of the scaling input on the output, let's contrast the scenario where we check the output when the input dataset is not scaled, with the output when the input dataset is scaled.

Input data is not scaled:

In the preceding table, note...

So far, in the dataset that we have considered, we have not looked at the distribution of values in the input dataset. Certain values of the input result in faster training. In this section, we will understand a scenario where weights are trained faster when the training time depends on the input values.

In this section, we will follow the model-building process in exactly the same way as we did in the previous section.

However, we will adopt a small change to our strategy:

• We will invert the background color, and also the foreground color. Essentially, the background will be colored white in this scenario, and the label will be written in...

In the previous sections, for all the models that we have built, we considered a batch size of 32. In this section, we will try to understand the impact of varying the batch size on accuracy.

To understand the reason batch size has an impact on model accuracy, let's contrast two scenarios where the total dataset size is 60,000:

• Batch size is 30,000
• Batch size is 32

When the batch size is large, the number of times of weight update per epoch is small, when compared to the scenario when the batch size is small.

The reason for a high number of weight updates per epoch when the batch size is small is that less data points are considered to calculate the loss value. This...

Until now, we have looked at model architectures where the neural network has only one hidden layer between the input and the output layers. In this section, we will look at the neural network where there are multiple hidden layers (and hence a deep neural network), while reusing the same MNIST training and test dataset that were scaled.

A deep neural network means that there are multiple hidden layers connecting the input to the output layer. Multiple hidden layers ensure that the neural network learns a complex non-linear relation between the input and output, which a simple neural network cannot learn (due to a limited number of hidden layers).

A typical...

So far, in the previous recipes, we used the default learning rate of the Adam optimizer, which is 0.0001.

In this section, we will manually set the learning rate to a higher number and see the impact of changing the learning rate on model accuracy, while reusing the same MNIST training and test dataset that were scaled in the previous recipes.

In the previous chapter on building feedforward neural networks, we learned that the learning rate is used in updating weights and the change in weight is proportional to the amount of loss reduction.

Additionally, a change in a weight's value is equal to the decrease in loss multiplied by the learning rate. Hence...

So far, in the previous recipes, we considered the loss optimizer to be the Adam optimizer. However, there are multiple other variants of optimizers, and a change in the optimizer is likely to impact the speed with which the model learns to fit the input and the output.

In this recipe, we will understand the impact of changing the optimizer on model accuracy.

To understand the impact of varying the optimizer on network accuracy, let's contrast the scenario laid out in previous sections (which was the Adam optimizer) with using a stochastic gradient descent optimizer in this section, while reusing the same MNIST training and test datasets that were scaled...

In some of the previous recipes, we have noticed that the training accuracy is ~100%, while test accuracy is ~98%, which is a case of overfitting on top of a training dataset. Let's gain an intuition of the delta between the training and the test accuracies.

To understand the phenomenon resulting in overfitting, let's contrast two scenarios where we compare the training and test accuracies along with a histogram of the weights:

• Model is run for five epochs
• Model is run for 100 epochs

The comparison-of-accuracy metric between training and test datasets between the two scenarios is as follows:

Once we plot the histogram of weights that are connecting the hidden layer to the output layer, we will notice that the 100...

In the previous section on the scaling dataset, we learned that optimization is slow when the input data is not scaled (that is, it is not between zero and one).

The hidden layer value could be high in the following scenarios:

• Input data values are high
• Weight values are high
• The multiplication of weight and input are high

Any of these scenarios can result in a large output value on the hidden layer.

Note that the hidden layer is the input layer to output layer. Hence, the phenomenon of high input values resulting in a slow optimization holds true when hidden layer values are large as well.

Batch normalization comes to the rescue in this scenario. We have already learned that, when input values are high, we perform scaling to reduce the input values. Additionally, we have learned that scaling can also be performed using...