Introducing penalized linear regression
Penalized regression models, such as ridge and lasso, are techniques that are used to handle problems such as multicollinearity, reduce overfitting, and even perform variable selection, especially when dealing with high-dimensional data with multiple input features.
Ridge regression (also called L2 regularization) is a method that adds a penalty equivalent to the square of the magnitude of coefficients. We would add this term to the loss function after weighting it by an additional hyperparameter, often denoted as λ, to control the strength of the penalty term.
Lasso regression (L1 regularization), on the other hand, is a method that, similar to ridge regression, adds a penalty for non-zero coefficients, but unlike ridge regression, it can force some coefficients to be exactly equal to zero when the penalty tuning parameter is large enough. The larger the value of the hyperparameter, λ, the greater the amount of shrinkage. The...
