Bayesian optimization

Leaving behind grid search (feasible only when the space of experiments is limited), the usual choice for the practitioner is to apply random search optimization or try a Bayesian optimization (BO) technique, which requires a more complex setup.

Originally introduced in the paper Practical Bayesian optimization of machine learning algorithms by Snoek, J., Larochelle, H., and Adams, R. P. (http://export.arxiv.org/pdf/1206.2944), the key idea behind Bayesian optimization is that we optimize a proxy function (also called a surrogate function) rather than the true objective function (which grid search and random search both do). We do this if there are no gradients, if testing the true objective function is costly (if it is not, then we simply go for random search), and if the search space is noisy and complex enough.

Bayesian search balances exploration with exploitation. At the start, it explores randomly, thus training the surrogate function as it goes...