Chapter 11: Monitoring Production Models with Amazon SageMaker Model Monitor and Clarify

Monitoring production machine learning (ML) models is a critical step to ensure that the models continue to meet business needs. Besides the infrastructure hosting the model, there are other important aspects of ML models that should be monitored regularly. As models age over a period of time, the real-world inference data distribution may change as compared to the data used for training the model. For example, consumer purchase patterns may change in the retail industry and economic conditions such as mortgage rates may change in the financial industry.

This gradual misalignment between the training and the live inference datasets can have a big impact on model predictions. Model quality metrics such as accuracy may degrade over time as well. Degraded model quality has a negative impact on business outcomes. Regulatory requirements, such as ensuring that ML models are unbiased and explainable...