Evaluating classification models

Imagine you are running a business with a large portfolio of customers, and you are trying to predict which customers are likely to stop using your service within the next year. This is a common binary classification model known as a customer churn model; many companies, whether banks, telecoms providers, insurance companies, or streaming services, can benefit from knowing which of their customers are most likely to churn so that they can take action to retain these customers.

You may have evaluated your customer churn model’s predictions on a test (holdout) set, for example, for the previous year, where you know whether a customer did, indeed, leave or stay with the company.

Important note

For this example, let’s refer to a customer who has churned as a “positive” outcome, as this is the outcome we are trying to predict (in this context, “positive” or “negative” does not have anything...