What Metric Should You Choose?

Another important aspect to consider on a serious note, is which metric we should consider while evaluating a model. There is no straightforward answer, as the best combination of metrics completely depend on the type of classification use case we are dealing with. One situation that commonly arises in classification use cases is imbalanced classes. It is not necessary for us to always have an equal distribution of positive and negative labels in data. In fact, in most cases, we would be dealing with a scenario where the positive class would be less than 30% of the data. In such cases, the overall accuracy would not be the ideal metric to consider.

Let's take a simple example to understand this better. Consider the example of predicting fraud in credit card transactions. In a realistic scenario, for every 100 transactions there may be just one or two fraud transactions. Now, if we use overall accuracy as the only metric to evaluate a model, even if we predict...