Building and publishing an ML pipeline
Let's go ahead and use all we have learned from the previous chapters and build a pipeline for data processing. We will use the Azure Machine Learning SDK for Python to define all the pipeline steps as Python code so that it can be easily managed, reviewed, and checked into version control as an authoring script.
We will define a pipeline as a linear sequence of pipeline steps. Each step will have an input and output defined as pipeline data sinks and sources. Each step will be associated with a compute target that defines both the execution environment as well as the compute resource for execution. We will set up an execution environment as a Docker container with all the required Python libraries and run the pipeline steps on a training cluster in Azure Machine Learning.
A pipeline runs as an experiment in your Azure Machine Learning workspace. We can either submit the pipeline as part of the authoring script, deploy it as a web service...