Improving generalization with regularization

Overfitting on the data is one of the biggest of machine learning. There are many machine learning algorithms that are able to train on the training data by remembering all cases. In this scenario, the algorithm might not be able to generalize and make a correct prediction on new data. This is an especially big threat for deep learning, where neural networks have large numbers of trainable parameters. Therefore, it is extremely important to create a representative validation set. 

Note

In deep learning, the general advice when tackling new problems is to overfit as much as you can on the training data first. This ensures that your model is able to train on the training data and is complex enough. Afterwards, you should regularize as much as you can to make sure the model is able to generalize on unseen data (the validation set) as well. 

Most of the techniques used to prevent overfitting can be placed under regularization. Regularization include...