Continuous model evaluation use cases
In this section, we will see some cases to demonstrate when model performance monitoring is essential.
To understand when continuous model evaluation is needed, let’s take an example case of a regression model to predict house prices:
- First, let’s take dummy example data showing house prices from January 2021 to July 2021.
When we deploy the regression model, we have to continuously monitor regression metrics such as Mean Square Error (MSA) and Mean Absolute Percentage Error (MAPE) to understand whether our model’s prediction started deviating a lot from the actual value. For example, let’s say that a regression model to predict house prices was trained in January using the data shown in the table in Figure 4.4:
Figure 4.4 – Mock house price data against features in Jan 2021 and July 2021
In the table in Figure 4.4, we have also shown the hypothetical price change...
