Revisiting hyperparameter tuning methods and packages
Throughout this book, we have discussed four groups of hyperparameter tuning methods, including exhaustive search, Bayesian optimization, heuristic search, and multi-fidelity optimization. All the methods within each group have similar characteristics to each other. For example, manual search, grid search, and random search, which are part of the exhaustive search group, all work by exhaustively searching through the hyperparameter space, and can be categorized as uninformed search methods.
Bayesian optimization hyperparameter tuning methods are categorized as informed search methods, where all of them work by utilizing both surrogate model and acquisition function. Hyperparameter tuning methods, which are part of the heuristic search group, work by performing trial and error. As for hyperparameter tuning methods from the multi-fidelity optimization group, they all utilize the cheap approximation of the whole hyperparameter...
