Hyperparameter Optimization

Hyperparameter optimization is the process of optimizing or finding the most optimal set of hyperparameters for a machine learning model. A hyperparameter is a parameter that defines the macro characteristics for a machine learning model. It is basically a metaparameter for the model. Hyperparameters are different from model parameters; model parameters are learned by the model during the learning process, however, hyperparameters are set by the data scientist designing the model and cannot be learned by the model.

To understand the concept more intuitively, let's explore the topic in layman terms. Consider the example of a decision tree model. The tree structure with the root node, decision nodes, and leaf nodes are (akin to the beta coefficients in logistic regression) are learned through training (fitting) of data. When the model finally converges (finds the optimal set of values for model parameters), we have the final tree structure that defines the traversal...