Holdout strategies
There are three aspects of a holdout strategy, and they can be mixed and matched to create many variations of the strategy. For instance, we might have a sampling strategy with a fixed split, a rolling window for the training data, and a recalibration of the model for each iteration. The three aspects are as follows:
- Sampling strategy: A sampling strategy is how we sample the validation split(s) from training data.
- Window strategy: A window strategy decides how we sample the window of training split(s) from training data.
- Calibration strategy: A calibration strategy decides whether a model should be recalibrated or not.
That said, designing a holdout validation strategy for a time series problem includes making decisions on these three aspects.
Sampling strategies are ways to pick one or more origins in the training data. These origins are points in time that determine the starting point of the validation split and the ending...
