Model validation
In Chapter 18, Evaluating Forecasts – Forecast Metrics, we learned about different forecast metrics that can be used to measure the quality of a forecast. One of the main uses for this is to measure how well our forecast is doing on test data (new and unseen data), but this comes after we train a model, tweak it, and tinker with it until we are happy with it. How do we know whether a model we are training or tweaking is good enough?
Model validation is the process of evaluating a trained model using data to assess how good the model is. We use the metrics we learned about in Chapter 18, Evaluating Forecasts – Forecast Metrics, to calculate the goodness of the forecast. But, there is one question we haven’t answered. Which part of the data do we use to evaluate? In a standard machine learning setup (classification or regression), we randomly sample a portion of the training data and call it validation data, and it is based on this data that all...
