Building and evaluating the ML model

Congratulations! You are now ready to train your model. You will first evaluate what set of algorithms will be a good fit for the given problem. Is it a regression or classification problem? How do you evaluate to see whether the model is achieving 75% correct predictability as described by the business?

Selecting evaluation criteria

Let's start with accuracy as the model evaluation criteria. This records how many times the predicted values are the same as the labels in the test dataset. However, if the dataset does not have the right variance, the model may guess the majority class for each example, which is effectively not learning anything about the minority class.

You decided to use the confusion matrix to see the accuracy for each class. Let's say you have 1,000 records in your data, out of which 50 are labeled as delayed. So, there are 950 examples with the on time label. Now, if the model correctly predicts 920 out of 950...