Regularizing with dropout

A widely used method for regularizing is dropout. Dropout is just randomly setting some neurons’ activations to zero during the training phase. Let’s first review how this works and then apply it to a multiclass classification task, the sklearn digits dataset, which is kind of an older and smaller version of the MNIST dataset.

Getting ready

Dropout is a widely adopted regularization approach in deep learning, due to its simplicity and effectiveness. The technique is easy to understand, yet can yield powerful results.

The principle is simple – during training, we randomly ignore some units by setting their activations to zero, as represented in Figure 7.10:

Figure 7.10 – On the left, a standard neural network with its connections, and, on the right, the same neural network with dropout, having, on average, 50% of its neurons ignored at training

Dropout adds one hyperparameter though: the dropout...