Chapter 3: Introduction to Machine Learning via Scikit-Learn

Activity 5: Generating Predictions and Evaluating the Performance of a Multiple Linear Regression Model

Solution:

1. Generate predictions on the test data using the following:

   predictions = model.predict(X_test)

   2.    Plot the predicted versus actual values on a scatterplot using the following code:

   import matplotlib.pyplot as plt

   from scipy.stats import pearsonr

    

   plt.scatter(y_test, predictions)

   plt.xlabel('Y Test (True Values)')

   plt.ylabel('Predicted Values')

   plt.title('Predicted vs. Actual Values (r = {0:0.2f})'.format(pearsonr(y_test, predictions)[0], 2))

   plt.show()

   Refer to the resultant output here:

   

   Figure 3.33: A scatterplot of predicted versus actual values from a multiple linear regression model

   Note

   There is a much stronger linear correlation between the predicted and actual values in the multiple linear regression model (r = 0.93) relative to the simple linear regression model (r = 0.62).

2. To plot the distribution of the residuals, refer to the code here:

   import seaborn as sns

   from scipy.stats import shapiro

    

   sns.distplot((y_test - predictions), bins = 50)

   plt.xlabel('Residuals')

   plt.ylabel('Density')

   plt.title('Histogram of Residuals (Shapiro W p-value = {0:0.3f})'.format(shapiro(y_test - predictions)[1]))

   plt.show()

   Refer to the resultant output here:

   

   Figure 3.34: The distribution of the residuals from a multiple linear regression model

   Note

   Our residuals are negatively skewed and non-normal, but this is less skewed than in the simple linear model.

3. Calculate the metrics for mean absolute error, mean squared error, root mean squared error, and R-squared, and put them into a DataFrame as follows:

   from sklearn import metrics

   import numpy as np

   metrics_df = pd.DataFrame({'Metric': ['MAE',

                                         'MSE',

                                         'RMSE',

                                         'R-Squared'],

                             'Value': [metrics.mean_absolute_error(y_test, predictions),

                                         metrics.mean_squared_error(y_test, predictions),

                                       np.sqrt(metrics.mean_squared_error(y_test, predictions)),

                                       metrics.explained_variance_score(y_test, predictions)]}).round(3)

   print(metrics_df)

   Please refer to the resultant output:

   

Figure 3.35: Model evaluation metrics from a multiple linear regression model

The multiple linear regression model performed better on every metric relative to the simple linear regression model.

Activity 6: Generating Predictions and Evaluating Performance of a Tuned Logistic Regression Model

Solution:

1. Generate the predicted probabilities of rain using the following code:

   predicted_prob = model.predict_proba(X_test)[:,1]

2. Generate the predicted class of rain using predicted_class = model.predict(X_test).
3. Evaluate performance using a confusion matrix and save it as a DataFrame using the following code:

   from sklearn.metrics import confusion_matrix

   import numpy as np

    

   cm = pd.DataFrame(confusion_matrix(y_test, predicted_class))

   cm['Total'] = np.sum(cm, axis=1)

   cm = cm.append(np.sum(cm, axis=0), ignore_index=True)

   cm.columns = ['Predicted No', 'Predicted Yes', 'Total']

   cm = cm.set_index([['Actual No', 'Actual Yes', 'Total']])

   print(cm)

   

   Figure 3.36: The confusion matrix from our logistic regression grid search model

   Note

   Nice! We have decreased our number of false positives from 6 to 2. Additionally, our false negatives were lowered from 10 to 4 (see in Exercise 26). Be aware that results may vary slightly.

4. For further evaluation, print a classification report as follows:

   from sklearn.metrics import classification_report

    

   print(classification_report(y_test, predicted_class))

Figure 3.37: The classification report from our logistic regression grid search model

By tuning the hyperparameters of the logistic regression model, we were able to improve upon a logistic regression model that was already performing very well.

Activity 7: Generating Predictions and Evaluating the Performance of the SVC Grid Search Model

Solution:

1. Extract predicted classes of rain using the following code:

   predicted_class = model.predict(X_test)

2. Create and print a confusion matrix using the code here:

   from sklearn.metrics import confusion_matrix

   import numpy as np

    

   cm = pd.DataFrame(confusion_matrix(y_test, predicted_class))

   cm['Total'] = np.sum(cm, axis=1)

   cm = cm.append(np.sum(cm, axis=0), ignore_index=True)

   cm.columns = ['Predicted No', 'Predicted Yes', 'Total']

   cm = cm.set_index([['Actual No', 'Actual Yes', 'Total']])

   print(cm)

   See the resultant output here:

   

   Figure 3.38: The confusion matrix from our SVC grid search model

3. Generate and print a classification report as follows:

   from sklearn.metrics import classification_report

    

   print(classification_report(y_test, predicted_class))

   See the resultant output here:

Figure 3.39: The classification report from our SVC grid search model

Here, we demonstrated how to tune the hyperparameters of an SVC model using grid search.

Activity 8: Preparing Data for a Decision Tree Classifier

Solution:

1. Import weather.csv and store it as a DataFrame using the following:

   import pandas as pd

    

   df = pd.read_csv('weather.csv')

2. Dummy code the Description column as follows:

   import pandas as pd

    

   df_dummies = pd.get_dummies(df, drop_first=True)

3. Shuffle df_dummies using the following code:

   from sklearn.utils import shuffle

    

   df_shuffled = shuffle(df_dummies, random_state=42)

4. Split df_shuffled into X and y as follows:

   DV = 'Rain'

   X = df_shuffled.drop(DV, axis=1)

   y = df_shuffled[DV]

5. Split X and y into testing and training data:

   from sklearn.model_selection import train_test_split

    

   X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

6. Scale X_train and X_test using the following code:

   from sklearn.preprocessing import StandardScaler

    

   model = StandardScaler()

   X_train_scaled = model.fit_transform(X_train)

   X_test_scaled = model.transform(X_test)

Activity 9: Generating Predictions and Evaluating the Performance of a Decision Tree Classifier Model

Solution:

1. Generate the predicted probabilities of rain using the following:

   predicted_prob = model.predict_proba(X_test_scaled)[:,1]

2. Generate the predicted classes of rain using the following:

   predicted_class = model.predict(X_test)

3. Generate and print a confusion matrix with the code here:

   from sklearn.metrics import confusion_matrix

   import numpy as np

    

   cm = pd.DataFrame(confusion_matrix(y_test, predicted_class))

   cm['Total'] = np.sum(cm, axis=1)

   cm = cm.append(np.sum(cm, axis=0), ignore_index=True)

   cm.columns = ['Predicted No', 'Predicted Yes', 'Total']

   cm = cm.set_index([['Actual No', 'Actual Yes', 'Total']])

   print(cm)

   Refer to the resultant output here:

   

   Figure 3.40: The confusion matrix from our tuned decision tree classifier model

4. Print a classification report as follows:

   from sklearn.metrics import classification_report

   print(classification_report(y_test, predicted_class))

   Refer to the resultant output here:

Figure 3.41: The classification report from our tuned decision tree classifier model

There was only one misclassified observation. Thus, by tuning a decision tree classifier model on our weather.csv dataset, we were able to predict rain (or snow) with great accuracy. We can see that the sole driving feature was temperature in Celsius. This makes sense due to the way in which decision trees use recursive partitioning to make predictions.

Activity 10: Tuning a Random Forest Regressor

Solution:

1. Specify the hyperparameter space as follows:

   import numpy as np

    

   grid = {'criterion': ['mse','mae'],

           'max_features': ['auto', 'sqrt', 'log2', None],

           'min_impurity_decrease': np.linspace(0.0, 1.0, 10),

           'bootstrap': [True, False],

           'warm_start': [True, False]}

2. Instantiate the GridSearchCV model, optimizing the explained variance using the following code:

   from sklearn.model_selection import GridSearchCV

   from sklearn.ensemble import RandomForestRegressor

    

   model = GridSearchCV(RandomForestRegressor(), grid, scoring='explained_variance', cv=5)

3. Fit the grid search model to the training set using the following (note that this may take a while):

   model.fit(X_train_scaled, y_train)

   See the output here:

   

   Figure 3.42: The output from our tuned random forest regressor grid search model

4. Print the tuned parameters as follows:

   best_parameters = model.best_params_

   print(best_parameters)

   See the resultant output below:

Figure 3.43: The tuned hyperparameters from our random forest regressor grid search model

Activity 11: Generating Predictions and Evaluating the Performance of a Tuned Random Forest Regressor Model

Solution:

1. Generate predictions on the test data using the following:

   predictions = model.predict(X_test_scaled)

2. Plot the correlation of predicted and actual values using the following code:

   import matplotlib.pyplot as plt

   from scipy.stats import pearsonr

    

   plt.scatter(y_test, predictions)

   plt.xlabel('Y Test (True Values)')

   plt.ylabel('Predicted Values')

   plt.title('Predicted vs. Actual Values (r = {0:0.2f})'.format(pearsonr(y_test, predictions)[0], 2))

   plt.show()

   Refer to the resultant output here:

   

   Figure 3.44: A scatterplot of predicted and actual values from our random forest regression model with tuned hyperparameters

3. Plot the distribution of residuals as follows:

   import seaborn as sns

   from scipy.stats import shapiro

    

   sns.distplot((y_test - predictions), bins = 50)

   plt.xlabel('Residuals')

   plt.ylabel('Density')

   plt.title('Histogram of Residuals (Shapiro W p-value = {0:0.3f})'.format(shapiro(y_test - predictions)[1]))

   plt.show()

   Refer to the resultant output here:

   

   Figure 3.45: A histogram of residuals from a random forest regression model with tuned hyperparameters

4. Compute metrics, place them in a DataFrame, and print it using the code here:

   from sklearn import metrics

   import numpy as np

   metrics_df = pd.DataFrame({'Metric': ['MAE',

                                         'MSE',

                                         'RMSE',

                                         'R-Squared'],

                             'Value': [metrics.mean_absolute_error(y_test, predictions),

                                       metrics.mean_squared_error(y_test, predictions),

                                       np.sqrt(metrics.mean_squared_error(y_test, predictions)),

                                       metrics.explained_variance_score(y_test, predictions)]}).round(3)

   print(metrics_df)

   Find the resultant output here:

Figure 3.46: Model evaluation metrics from our random forest regression model with tuned hyperparameters

The random forest regressor model seems to underperform compared to the multiple linear regression, as evidenced by greater MAE, MSE, and RMSE values, as well as less explained variance. Additionally, there was a weaker correlation between the predicted and actual values, and the residuals were further from being normally distributed. Nevertheless, by leveraging ensemble methods using a random forest regressor, we constructed a model that explains 75.8% of the variance in temperature and predicts temperature in Celsius + 3.781 degrees.