Scaling up as a function of world size with SageMaker
In this section, we’ll break down two critical concepts that you need to master hyperparameter tuning, especially in the context of distributed training. The first one is the concept of scaling, especially using hyperparameter tuning as a method to run smaller experiments before ultimately running your large training job. The second is using tips and tricks available on SageMaker for hyperparameter tuning generally.
Tuning on a sample of your data and updating based on world size
As you’ve learned in this chapter, hyperparameter tuning is a great way to eke out performance gains, but it can require intensive compute that executes a large number of experiments. You might be wondering, How do I easily apply this to my use case with a dataset size of at least a few hundred GB, and maybe a few TB or more? The answer is just to start with a tiny sample!
The goal of tuning at a tiny fraction of your dataset is to...
