Using pipelines in ML workflows
Separating your workflow into reusable configurable steps and combining these steps into an end-to-end pipeline provides many benefits for implementing end-to-end ML processes. Multiple teams can own and iterate on individual steps to improve the pipeline over time, while others can easily integrate each version of the pipeline into their current setup.
The pipeline itself doesn't only split code from execution; it also splits the execution from orchestration. Hence, you can configure individual compute targets that can be used to optimize your execution and provide parallel execution while you don't have to touch the ML code.
We will take a quick look at Azure Machine Learning pipelines and why they are your tool of choice when implementing ML workflows in Azure. In the following section, Building and publishing an ML pipeline, we will dive a lot deeper and explore the individual features by building such a pipeline.