Building for modularity
By now, you should be familiar with the layers of our Pragmatic Data Platform (PDP) and the core principles of each layer:
Figure 13.1: The layers of the Pragmatic Data Platform
Let’s quickly recap them:
- Storage layer: Here, we adapt incoming data to how we want to use it without changing its semantics and store all the data: the good, the bad, and the ugly.
The core principle is to isolate here the platform state – that is, models that depend on previous runs such as snapshots or incremental models – so that the next layers can be rebuilt from scratch on top of the storage layer.
The perspective is source centric, with one load pipeline for each source table.
- Refined layer: Here, we apply master data and implement business rules.
The core principle here is to apply modularity while building more abstract and general-use concepts on top of the simpler, source-centric ones from...
