Delineating hyperparameter search algorithms

In this section, we will take a closer look at three algorithms that traverse the hyperparameter search space. These algorithms are implemented by the tf.keras API.

Hyperband

Hyperparameter search is an inherently tedious process that requires a budget B to test a finite set of possible hyperparameter configurations n. In this context, budget simply means compute time as indicated by the epoch, and the training data subsets. The hyperband algorithm takes advantage of early stopping and successive halving so that it can evaluate more hyperparameter configurations in a given time and with a given set of hardware resources. Early stopping helps eliminate underperforming configurations before too much training time is invested in them.

The successive halving method is very intuitive: for a set of hyperparameter configurations, run them through the same budget (that is, epoch, memory, and training data subset size). Then we rank the...