Configuring data destinations

Once the batch and streaming ingestion pipelines have been invoked, they will fetch data from the data sources and dump the results into the data destination. The data destination for the bronze layer is Azure Data Lake Storage Gen2:

1. We will now use the Azure client to create an Azure Data Lake Storage account. Copy each of the following commands, line by line, and paste them inside the Cloud Shell window. Then, press Enter:

   STORAGEACCOUNTNAME="traininglakehouse"
   RESOURCEGROUPNAME="training_rg"
   LOCATION="eastus"
   az storage account create --name $STORAGEACCOUNTNAME --resource-group $RESOURCEGROUPNAME --kind StorageV2 --location $LOCATION  --hns true --sku Standard_LRS --tags owner=data engineering project=lakehouse environment=development

   If the preceding commands are successful, you should see an output that looks like this:

   Figure 5.19 – Output of the creation of the Data Lake Storage account

2. You...