Regularization
Regularization is a collection of techniques that can be used to prevent overfitting. Regularization adds information, often in the form of a penalty against complexity, to a problem. Occam's razor states that the hypothesis with the fewest assumptions is best. Accordingly, regularization attempts to find the simplest model that explains the data.
scikit-learn provides several regularized linear regression models. Ridge regression, also known as Tikhonov regularization, penalizes model parameters that become too large. Ridge regression modifies the RSS cost function by adding the L2 norm of the coefficients, as follows:
Lambda is a hyperparameter that controls the strength of the penalty. Recall from Chapter 3, Classification and Regression with K-Nearest Neighbors, that hyperparameters are parameters of the model that control how the learning algorithm learns. As lambda increases, the penalty increases, and the value of the cost function increases. When lambda is equal to zero...
