Ingesting Analytical Data
Analytical data is a bundle of data that serves various areas (such as finances, marketing, and sales) in a company, university, or any other institution, to facilitate decision-making, especially for strategic matters. When transposing analytical data to a data pipeline or a usual Extract, Transform, and Load (ETL) process, it corresponds to the final step, where data is already ingested, cleaned, aggregated, and has other transformations accordingly to business rules.
There are plenty of scenarios where data engineers must retrieve data from a data warehouse or any other storage containing analytical data. The objective of this chapter is to learn how to read analytical data and its standard formats and cover practical use cases related to the reverse ETL concept.
In this chapter, we will learn about the following topics:
- Ingesting Parquet files
- Ingesting Avro files
- Applying schemas to analytical data
- Filtering data and handling...
