Measuring the Performance of Data Movement

This section equips you with the tools and techniques to assess how data flows within pipelines. By monitoring and analyzing data movement metrics, you can identify bottlenecks, ensure timely delivery, and maintain data integrity. In this section, you will explore the key concepts, best practices, and practical approaches to measuring and enhancing data movement efficiency.

Note

This section primarily focuses on the Measure performance of data movement concept of the DP-203: Data Engineering on Microsoft Azure exam.

ADF provides a rich set of performance metrics under its Monitoring tab. Say you have a sample Copy Data activity as part of a pipeline called FetchDataFromBlob, which copies data from Blob Storage into Azure Data Lake Storage Gen2 (ADLS Gen2). In order to see the details of each of the pipelines, click on the Pipeline runs option under the Monitoring tab, as shown in Figure 9.13. To check the diagnostic settings in ADF...