Model selection
We looked at five different models in examining this dataset. The following points were the test set error of these models:
- Best subsets is 0.51
- Ridge regression is 0.48
- LASSO is 0.44
- Elastic net is 0.48
- LASSO with CV is 0.46
On a pure error, LASSO with seven features performed the best. However, does this best address the question that we are trying to answer? Perhaps the more parsimonious model that we found using CV with a lambda of ~0.165 is more appropriate. My inclination is to put forth the latter as it is more interpretable.
Having said all this, there is clearly a need for domain-specific knowledge from oncologists, urologists, and pathologists in order to understand what would make the most sense. There is that, but there is also the need for more data. With this sample size, the results can vary greatly just by changing the randomization seeds or creating different train and test sets. (Try it and see for yourself.) At the end of the day, these results may likely raise...
