Say we want to predict whether someone will catch a viral disease. We can then build a classifier to predict whether they will catch the viral infection or not. Nevertheless, when the percentage of those who may catch the infection is too low, the classifier's binary predictions may not be precise enough. Thus, with such uncertainty and limited resources, we may want to only put in quarantine those with more than a 90% chance of catching the infection. The classifier's predicted probability sounds like a good source for such estimation. Nevertheless, we can only call this probability reliable if 9 out of 10 of the samples we predict to be in a certain class with probabilities above 90% are actually in this class. Similarly, 80% of the samples with probabilities above 80% should also end up being in...
Calibrating a classifier's probabilities
"Every business and every product has risks. You can't get around it."
– Lee Iacocca
