The functional steps in DataOps

In Part 2, The Functional Steps in DataOps, we saw how within a DataOps project, we take five overarching steps:

1. Sourcing the data
2. Data processing and transformations
3. Destination management
4. Value extraction
5. Beginning advanced analytics

We detailed each step in a separate chapter. We also split the value extraction step into two chapters, allowing for the getting primary value extraction details and extending our analytic capabilities with the advanced spatial and machine learning features in Alteryx.

Sourcing the data

Chapter 4, Sourcing the Data, began our data pipeline construction by connecting to three different data source location types:

• Internal organization data sources
• Downloading public files
• Using public APIs for data extraction

These three data locations provide the basis for most of the data sources you find as a data engineer. The internal data sources are any files or databases...