Holdout strategies
There are three aspects of a holdout strategy, and they can be mixed and matched to create many variations of the strategy. The three aspects are as follows:
- Sampling strategy – A sampling strategy is how we sample the validation split(s) from training data.
- Window strategy – A window strategy decides how we sample the window of training split(s) from training data.
- Calibration strategy – A calibration strategy decides whether a model should be recalibrated or not.
That said, designing a holdout validation strategy for a time series problem includes making decisions on these three aspects.
Sampling strategies are ways to pick one or more origins in the training data. These origins are point(s) in time that determine the starting point of the validation split and the ending point of the training split. The exact length of the validation split is governed by a parameter 
, which is the horizon chosen for validation. The...
