Building and publishing an ML pipeline

Let's go ahead and use our knowledge from the previous chapters to build a pipeline for data processing. We will use the Azure Machine Learning Python SDK to define all pipeline steps as Python code so the pipeline can be easily managed, reviewed, and checked into version control as an authoring script.

We will define a pipeline as a linear sequence of steps. Each step will have an input and output defined as pipeline data sinks and sources. Each step will be associated with a compute target that defines both the execution environment and the compute resource for execution. We will set up an execution environment as a Docker container with all the required Python libraries and run the pipeline steps on a training cluster in Azure Machine Learning.

A pipeline runs as an experiment in your Azure Machine Learning workspace. We can either submit the pipeline as part of the authoring script, deploy it as web service and trigger it through...