Summary

In this chapter, we covered metrics as well as sampling techniques for classification ML algorithms. Once we have a few models that we've tried fitting to the data, we can compare them with metrics. For evaluating metrics, we want to split our data into training/validation or training/test splits or use cross-validation to compare models, using the test or validation set to compute our metrics. Common classification metrics include accuracy, precision, recall, F1 score, and the AUC score from the ROC curve, and we can also use Cohen's Kappa as a metric. Additionally, we can optimize the rounding threshold for probability predictions from models using a few different methods with Youden's J, using the ROC curve or maximizing the F1 score. Visualizations for many of these metrics as well as confusion matrices can be created with the sklearn, mlxtend, and yellowbrick packages.

Lastly, we looked at undersampling and oversampling data to achieve a balance between...