Transforming deep learning datasets at scale on AWS
At this point, you must be thinking now I know how to build and test my data loader, and even put my data on FSx for Lustre to integrate with SageMaker training, but what if I need to do large-scale downloads or transformations ahead of time? How can I do those at a large scale, in a cost-effective and simple way?
While there are many different tools and perspectives for attacking this problem, my personal favorite is always to take the simplest, least expensive, and most scalable approach. To me, that’s actually with job parallelism on SageMaker Training.
As it turns out, SageMaker Training is a very broad compute service offering you can use to run essentially any type of script. In particular, you can use it to run large CPU-based data transformation jobs in parallel. There’s no upper limit on how many SageMaker Training jobs you can run, and we have customers who run thousands of jobs a day in order to train...
