Using Regularization for Feature Selection

In the previous section, we saw how an evaluation metric such as the RMSE can be used to decide whether a variable should be included in a model or not. However, this method can be cumbersome when there are many variables involved.

When a model contains extraneous variables (variables that are not related to the outcome of interest), it can become more difficult to interpret the model. It can also lead to overfitting, where the model may change drastically if you use a different subset of the data to train the model. Therefore, it is important to select only those features that are related to the outcome for training the model.

One common way to select which features will be used by a model is to use regularization. The idea of regularization is that the model will be asked not only to try to predict the training points as accurately as possible, but will have the additional constraint of trying to minimize the weight that it puts on each of the variables...