Orchestrating ETL operations with Data Factory pipelines

Data Factory pipelines are an extremely useful tool in analytics projects. Earlier in this chapter, you created a data pipeline that runs a Copy data activity to load the stage.DimCity table. This data was later used in a stored procedure to load a dimensional model table called dbo.DimCity. It is now time to extend the pipeline to orchestrate the entire ETL process, which will include the following:

1. Dropping and recreating all stage schema tables so that data is not duplicated from prior runs
2. Loading the stage.DimCity table using the Copy data activity
3. Loading the stage.DimDate and stage.FactSale tables using the T-SQL COPY command by executing a stored procedure
4. Incrementally loading the dimensional model using a stored procedure

Let’s extend the pipeline created earlier in the chapter:

1. Return to the pipeline created in the Loading data section earlier in the chapter by navigating to...