Developing the master pipeline

To start the creation of the master pipeline, make sure you have the electroniz_batch_ingestion_pipeline, electroniz_curation_pipeline, and electroniz_aggregation_pipeline pipelines installed and in working condition in the Azure data factory:

Figure 9.3 – The Electroniz ingestion, curation, and aggregation pipelines

We will get started by performing the following steps:

1. Using the Azure portal, navigate to Home > All Resources > trainingdatafactory100. Click on Open underneath Open Azure Data Factory Studio.
2. Using the panel on the left-hand side, from the menu, click on Author. In the Factory Resources panel, click on the three dots to the right of Pipelines and choose New Pipeline.

   In the Properties panel, input the following:

   • Name: electroniz_master_pipeline
   • Description: This pipeline runs the ingestion, curation, and aggregation pipeline.

   Using the bottom panel, click on Parameters and add the following...