Linear regression API with ridge regression and 'auto' optimization selection in Spark 2.0

In this recipe, we implement ridge regression the LinearRegression interface. We use the elastic net parameter to set the appropriate value to a full L2 penalty, which in turn selects the ridge regression accordingly.

How to do it...

1. We use a housing data set from the UCI machine library depository.

2. Download the entire data set from the following URLs:
   • https://archive.ics.uci.edu/ml/datasets/Housing
   • https://archive.ics.uci.edu/ml/machine-learning-databases/housing/

The dataset is comprised of 14 columns with the first 13 columns being the independent variables (that is, features) that try to explain the median price (that is, last column) of an owner-occupied house in Boston, USA.

We have chosen and cleaned the first eight columns as features. We use the first 200 rows to train and predict the median price:

1. • CRIM: Per capita crime rate by town
   • ZN: Proportion of residential land zoned for lots over 25,000 sq...