Managing data and datasets in the cloud

When you run an ML experiment or pipeline on your local development machine, you often don't need to manage your datasets as they are stored locally. However, as soon as you start training an ML model on remote compute targets, such as a VM in the cloud, you must make sure that the script can access the training data. And if you deploy a model that requires a certain dataset during scoring—for example, the lookup data for labels and the like—then this environment needs to access the data as well. As you can see, it makes sense to abstract the datasets for an ML project, both from the point of view of physical access and access permissions.

First, we will show how you can create a data store object to connect the Azure Machine Learning workspace to other data services, such as blob or file storage, data lake storage, and relational data stores, such as SQL Server and PostgreSQL. Once a data store is attached, we can register...