Questions

1. Can a well-calibrated model have low accuracy? What about the reverse: can a model with high accuracy be poorly calibrated?
2. Take a limited classification dataset with, say, only 100 data points. Train a decision tree model using this dataset and then assess its calibration.
   1. Calibrate the model using Platt’s scaling. Measure the Brier score after calibration.
   2. Calibrate the model using isotonic regression. Measure the Brier score after calibration
   3. How do the Brier scores differ in (A) and (B)?
   4. Measure the AUC, accuracy, precision, recall, and F1 score of the model before and after calibration.
3. Take a balanced dataset, say with 10,000 points. Train a decision tree model using it. Then check how calibrated it is.
   1. Calibrate the model using Platt’s scaling. Measure the Brier score after calibration.
   2. Calibrate the model using isotonic regression. Measure the Brier score after calibration.
   3. How do the Brier scores differ in (a) and (b)?
   4. Measure the AUC, accuracy...