Process of creating a data pipeline

In analytics-centric organizations, it is very common to have multiple data pipelines, each one addressing a different use case. To make matters worse, each use case may be owned by a different sub-group within the organization and require a different dataset. In such cases, it becomes extremely important to carefully plan and design the data pipeline operation so that efficiencies can be discovered and repetitive work can be avoided. The creation of data pipelines is done in phases. In the subsequent sections, we will learn about each phase separately.

Before we deep dive into the details, the following diagram is important to highlight how each phase stacks on top of the other. The most important thing to notice in this diagram is that if data engineers diligently follow the recommended actions for each phase, the workload for each phase significantly decreases, and success is virtually guaranteed:

Figure 4.2 –...