Applying changes (CDC) to Delta tables with Delta Live Tables

Delta Live Tables also supports CDC, a technique that’s used to identify and capture changes made to data in a source database and then deliver those changes in real time to a target system. CDC enables you to keep your data lake or data warehouse in sync with your operational databases and also supports real-time analytics and data science.

One of the challenges of CDC is how to handle slowly changing dimensions (SCDs), which are dimensions that store and manage both current and historical data over time in a data warehouse. For example, a customer dimension may have attributes such as name, address, and phone number that can change over time. Depending on your business requirements, you may want to track the history of these changes in different ways. Several types of SCDs define how to handle these changes, such as Type 1 (overwrite), Type 2 (add new row), Type 3 (add new column), and so on.

In this recipe...