Defining drift
It is a well-known and commonly observed problem that models tend to start performing worse with time. Whether your metric is accuracy, R2 score, F1 score, or anything else, you will see a slow but steady decrease in performance over time if you put models into production and do not update them.
Depending on your use case, this decrease may become problematic quickly or slowly. Some use cases need to have continuous, near-perfect predictions. In some use cases— for example, for specialized ML in which the models have a direct impact on life—you would be strongly shocked if you observed a 1 percent decrease. In other use cases, ML is used more as automation than as prediction, and the business partners may not even notice a 5 percent decrease.
Whether it is going to impact you is not the question here. What is sure, is that in general, you will see your models decreasing. The goal for this chapter is to make sure to find out why model performance is...
