Designing and Implementing Incremental Loads

Incremental loading (or delta loading) refers to the process of loading smaller increments of data into a storage solution. For example, you could have daily data that is being loaded into a data lake or hourly data flowing into an Extract, Transform, Load (ETL) pipeline, and so on. During data-ingestion scenarios, it is very common to do a bulk upload followed by scheduled incremental loads.

Note

This section primarily focuses on the Design and implement incremental loads concept of the DP-203: Data Engineering on Microsoft Azure exam.

Azure has a very versatile service called ADF, which can help with incremental loading. Since this is the first time you are using ADF in this book, you need to learn a little more about it now, as the information will come in handy in future chapters.

ADF is a managed cloud service that can be used to coordinate and orchestrate complex cloud- or hybrid- (a mixture of cloud and on-premises) based...