Choosing the right regularization

Linear models share this regularization method with L1 and L2 penalization. The only difference in the implementation is the fact that linear regression has its own class for each regularization type, as mentioned here:

• LinearRegression for no regularization
• RidgeRegression for L2 regularization
• Lasso for L1 regularization
• ElasticNet for both L1 and L2

Logistic regression has an integrated implementation, passing L1 or L2 as the class parameter.

Note

With Support Vector Machines (SVMs), the scikit-learn’s implementation provides a C parameter for L2 regularization for both the SVC classification class and the SVR regression class.

But for linear regression as well as logistic regression, one question remains: should we use L1 or L2 regularization?

In this recipe, we will provide some practical tips about whether to use L1 or L2 penalization in some cases and then we will perform a grid search on the breast...