Selecting model performance metrics

The most relevant question about any model is, How well does it predict? Regardless of any other positive properties that a model may possess, models that don't predict well are just not very useful. How to best measure predictive performance depends both on the specific problem being solved and the choices available to the data scientist. H2O provides multiple options for measuring model performance.

For measuring predictive model performance in regression problems, H2O provides R2, mean squared error (MSE), root mean squared error (RMSE), root mean squared logarithmic error (RMSLE), and mean absolute error (MAE) as metrics. MSE and RMSE are good default options, with RMSE being our preference because the metric is expressed in the same units as the predictions (rather than squared units, as in the case of MSE). All metrics based on squared error are sensitive to outliers in general. If robustness to outliers is a requirement, then MAE is...