Copying data from a SQL Server virtual machine to a data lake using the Copy data wizard

A common scenario in data engineering projects is where you need to ingest data from a relational database engine such as SQL Server, Oracle, or MySQL to a data lake. This recipe will show you how to ingest data from SQL Server, which has been installed in an Azure VM, to an Azure Data Lake. This method will work in on-premises SQL Server to Azure Data Lake instances too, but you will need to install an integration runtime. This will be covered in the next chapter. In this recipe, we will focus on copying data from SQL Server in an Azure VM to a data lake. We will be using the user-friendly Copy data wizard to transfer the data.

Getting ready

Follow these steps:

1. Provision Azure Data Factory, as explained in the Provisioning Azure Data Factory recipe.
2. Log in to Azure PowerShell using the Connect-AzAccount command and provision the blob storage account and container, as shown in...