Building for modularity

By now, you should be familiar with the layers of our Pragmatic Data Platform (PDP) and the core principles of each layer:

Figure 13.1: The layers of the Pragmatic Data Platform

Let’s quickly recap them:

• Storage layer: Here, we adapt incoming data to how we want to use it without changing its semantics and store all the data: the good, the bad, and the ugly.

The core principle is to isolate here the platform state – that is, models that depend on previous runs such as snapshots or incremental models – so that the next layers can be rebuilt from scratch on top of the storage layer.

The perspective is source centric, with one load pipeline for each source table.

• Refined layer: Here, we apply master data and implement business rules.

The core principle here is to apply modularity while building more abstract and general-use concepts on top of the simpler, source-centric ones from...