Grid testing

Let us consider a typical classification problem. Assume you have a dataset and you divide it into training (T) and validating (V) datasets. Here you are trying to solve an optimization problem, let's say P, in which one is trying to reduce the training error in addition to regularization terms, where the optimization problem is a function of model parameter m, training sample T, and some hyperparameters and . Solving for given  and  gives you the value of parameter m. Now one can apply the estimated parameters on the validation sample to get the validation error function, and optimize it to get the set of  and  to minimize the error function. But this optimization problem will be very expensive as, for each set of  and , you need to optimize the objective function, which might not be a convex, concave, or smooth function.

So we subset the set of  and  and, for each pair of selected  and , we solve the optimization problem. This looks like a grid in space so we call it a grid...