Assume for a moment that we implemented the cross-validation procedure for two versions of our k-NN classifier. The resulting test scores are-- 92.34% for Model A, and 92.73% for Model B. How do we know which model is better?
Following our logic introduced here, we might argue for Model B because it has a better test score. But what if the two models are not significantly different? These could have two underlying causes, which are both a consequence of the randomness of our testing procedure:
- For all we know, Model B just got lucky. Perhaps we chose a really low k for our cross-validation procedure. Perhaps Model B ended up with a beneficial train-test split so that the model had no problem classifying the data. After all, we didn't run tens of thousands of iterations like in bootstrapping to make sure the result holds in general...
