Ridge, Lasso, and ElasticNet

In this section, we are going to analyze the most common regularization methods and how they can impact the performance of a linear regressor. In real-life scenarios, it's very common to work with dirty datasets, containing outliers, inter-dependent features, and different sensitivity to noise. These methods can help the data scientist mitigate the problems, yielding more effective and accurate solutions.

Ridge

Ridge regression (also known as Tikhonov regularization) imposes an additional shrinkage penalty to the ordinary least squares cost function to limit its squared L₂ norm:

X is a matrix containing all samples as rows and the term θ represents the weight vector. The additional...