Monitoring models in production
Our work isn’t over once we’ve developed and deployed a model – we need to track the model’s performance and its overall health over time and make adjustments if we observe that the model performance deteriorates. In this section, we’ll discuss some of the characteristics of our models that we typically need to monitor.
Objective model performance
Not surprisingly, the most prominent aspect of our model that we need to monitor is how it performs concerning the objective it was created to achieve. We discussed objective metrics in previous chapters of this book, such as Mean Squared Error (MSE), Accuracy, F1 score, and AUC-ROC, among others. An example of AUC-ROC is depicted in Figure 10.5 for reference:
Figure 10.5: AUC-ROC
Objective metrics tell us how our model is performing in terms of the main purpose our model is intended to serve, such as predicting housing prices or identifying...
