L1 and L2 Regularization
The most common type of regularization for deep learning models is the one that keeps the weights of the network small. This type of regularization is called weight regularization and has two different variations: L2 regularization and L1 regularization. In this section, you will learn about these regularization methods in detail, along with how to implement them in Keras. Additionally, you will practice applying them to real-life problems and observe how they can improve the performance of a model.
L1 and L2 Regularization Formulation
In weight regularization, a penalizing term is added to the loss function. This term is either L2 norm (the sum of the squared values) of the weights, or L1 norm (the sum of the absolute values) of the weights. If L1 norm is used, then it will be called L1 regularization. If L2 norm is used, then it will be called L2 regularization. In each case, the sum is multiplied by a hyperparameter called a regularization parameter (lambda).
Therefore...