Hyperparameter tuning with Hyperopt

The Azure Databricks Runtime for Machine Learning includes Hyperopt, a Python library that is intended to be used on distributed computing systems to facilitate the learning process for an optimal set of hyperparameters. At its core, it's a library that receives a function that we need to either minimize or maximize, and a set of parameters that define the search space. With this information, Hyperopt will explore the search space to find the optimal set of hyperparameters. Hyperopt uses a stochastic search algorithm to explore the search space, which is much more efficient than using a traditional deterministic approach such as random search or grid search.

Hyperopt is optimized for use in distributed computing environments and provides support for libraries such as PySpark MLlib and Horovord, the latter of which is a library for distributed deep learning training that we will focus on later. It can also be applied in single-machine environments...