Identifying and correcting poor performances

A performant predictive model is one that produces reliable and satisfying predictions on new data. There are two situations where the model will fail to consistently produce good predictions, and both depend on how the model is trained. A poorly trained model will result in underfitting, while an overly trained model will result in overfitting.

Underfitting

Underfitting means that the model was poorly trained. Either the training dataset did not have enough information to infer strong predictions, or the algorithm that trained the model on the training dataset was not adequate for the context. The algorithm was not well parameterized or simply inadequate for the data.

If we measure the prediction error not only on the validation set but also on the training set, the prediction error will be large if the model is underfitting. Which makes sense: if the model cannot predict the training, it won't be able to predict the outcomes in the validation set...