An alternative procedure to k-fold cross-validation is bootstrapping.
Instead of splitting the data into folds, bootstrapping builds a training set by drawing samples randomly from the dataset. Typically, a bootstrap is formed by drawing samples with replacement. Imagine putting all of the data points into a bag and then drawing randomly from the bag. After drawing a sample, we would put it back in the bag. This allows for some samples to show up multiple times in the training set, which is something cross-validation does not allow.
The classifier is then tested on all samples that are not part of the bootstrap (the so-called out-of-bag examples), and the procedure is repeated a large number of times (say, 10,000 times). Hence, we get a distribution of the model's score that allows us to estimate the robustness of the model.
...
