What this book covers
Chapter 1, Getting Started with ADF, will provide a general introduction to the Azure data platform. In this chapter, you will learn about the ADF interface and options as well as common use cases. You will perform hands-on exercises in order to find ADF in the Azure portal and create your first ADF job.
Chapter 2, Orchestration and Control Flow, will introduce you to the building blocks of data processing in ADF. The chapter contains hands-on exercises that show you how to set up linked services and datasets for your data sources, use various types of activities, design data-processing workflows, and create triggers for data transfers.
Chapter 3, Setting Up Synapse Analytics, covers key features and benefits of cloud data warehousing and Azure Synapse Analytics. You will learn how to connect and configure Azure Synapse Analytics, load data, build transformation processes, and operate data flows.
Chapter 4, Working with Data Lake and Spark Pools, will cover the main features of the Azure Data Lake Storage Gen2. It is a multimodal cloud storage solution that is frequently used for big data analytics. We will load and manage the datasets that we will use for analytics in the next chapter.
Chapter 5, Working with Big Data and Databricks, will actively engage with analytical tools from Azure’s data services. You will learn how to build data models in Delta Lake using Azure Databricks and mapping data flows. Also, this recipe will show you how to set up HDInsights clusters and how to work with delta tables.
Chapter 6, Data Migration – Azure Data Factory and Other Cloud Services, will walk though several illustrative examples on migrating data from Amazon Web Services and Google Cloud providers. In addition, you will learn how to use ADF’s custom activities to work with providers who are not supported by Microsoft’s built-in connectors.
Chapter 7, Extending Azure Data Factory with Logic Apps and Azure Functions, will show you how to harness the power of serverless execution by integrating some of the most commonly used Azure services: Azure Logic Apps and Azure Functions. These recipes will help you understand how Azure services can be useful in designing Extract, Transform, Load (ETL) pipelines.
Chapter 8, Microsoft Fabric and Power BI, Azure ML, and Cognitive Services, will teach you how to build an ADF pipeline that operates on a pre-built Azure ML model. You will also create and run an ADF pipeline that leverages Azure AI for text data analysis. In the last three recipes, you’ll familiarize yourself with the primary components of Microsoft Fabric Data Factory.
Chapter 9, Managing Deployment Processes with Azure DevOps, will delve into setting up CI and CD for data analytics solutions in ADF using Azure DevOps. Throughout the process, we will also demonstrate how to use Visual Studio Code to facilitate the deployment of changes to ADF.
Chapter 10, Monitoring and Troubleshooting Data Pipelines, will introduce tools to help you manage and monitor your ADF pipelines. You will learn where and how to find more information about what went wrong when a pipeline failed, how to debug a failed run, how to set up alerts that notify you when there is a problem, and how to identify problems with your integration runtimes.
Chapter 11, Working with Azure Data Explorer, will help you to set up a data ingestion pipeline from ADF to Azure Data Explorer: it includes a step-by-step guide to ingesting JSON data from Azure Storage and will teach you how to transform data in Azure Data Explorer with ADF activities.
Chapter 12, The Best Practices of Working with ADF, will guide you through essential considerations, strategies, and practical recipes that will elevate your ADF projects to new heights of efficiency, security, and scalability.
