Packages and settings – R and Python

We will need four R packages in ridge, DAAG, splines, and MASS. The required Python packages are matplotlib, pandas, numpy, pylab, statsmodels, and sklearn:

1. First set the working directory in R:

   setwd("MyPath/R/Chapter_06")

2. Load the essential R packages:

   > library(RSADBE)
   > library(ridge)
   > library(DAAG)
   > library(splines)
   > library(MASS)

3. Set the working directory and required packages and functions in Python now:

   

Using these packages and functions, we will be able to carry out the computations required in the rest of the chapter.

The overfitting problem

The limitation of the linear regression model is best understood through an example. I have created a hypothetical dataset for understanding the problem of overfitting. A scatterplot of the dataset is shown in the figure, A non-linear relationship displayed by scatter plot.

It appears from the scatterplot that, for x-values up to 6, there is a linear increase in y, and an eye-bird estimate of the...