Chapter 4: Sourcing the Data
The first step of creating a new data pipeline is the process of sourcing the raw dataset. While scoping and defining the dataset are crucial parts of the entire data pipeline project, the framework for extracting the information is well established in general project management and the underlying Agile framework. Therefore, in this chapter, we will begin at the point of having the initial requirements defined and understood.
We will focus on the methods for accessing data sources from both internal sources, freely available public sources, and application programming interfaces (APIs) that have security applied.
We will also discuss some methods for validating the data sources you connect to and ensuring that the raw data structure has not changed. If the data source has changed, we will have automated methods to assess those changes.
In this chapter, we will cover the following topics:
- How to connect to different internal data sources...
