Model validation
In Chapter 19, Evaluating Forecast Errors—A Survey of Forecast Metrics, we learned about different forecast metrics that can be used to measure the quality of a forecast. One of the main uses for this is to measure how well our forecast is doing on test data (new and unseen data), but this comes after we train a model, tweak it, and tinker with it until we are happy with it. How do we know whether a model we are training or tweaking is good enough?
Model validation is the process of evaluating a trained model using data to assess how good the model is. We use the metrics we learned about in Chapter 19, Evaluating Forecast Errors—A Survey of Forecast Metrics, to calculate the goodness of the forecast. But there is one question we haven’t answered. Which part of the data do we use to evaluate? In a standard machine learning setup (classification or regression), we randomly sample a portion of the training data and call it validation data, and...
