Copying files to a database from a data lake using a control flow and copy activity

In this recipe, we will be building a pipeline that will copy a group of files in blob storage to Azure SQL Database, but only if the filenames contain today’s date as a suffix. Follow these steps:

1. Get the list of files to be copied using the Get Metadata activity in the data factory.
2. Use the Filter activity to filter the file whose suffix is the current date.
3. Use the ForEach activity to loop through the files.
4. Use the Copy activity to load the file into Azure SQL Database.

Getting ready

Follow these steps:

1. Provision a data factory, as explained in the Provisioning Azure Data Factory recipe of this chapter.
2. Log in to Azure PowerShell using Connect-AzAccount.
3. Execute the following command to create a storage account and a container:

   $storageaccountname="packtadeadfadl"
   $resourcegroup="PacktADEADF"
   $containername="dataloading...