A Data Engineer’s Journey – Background Challenges

The purpose of this chapter is to explain the challenges you must face to navigate a successful journey as a data engineer. The intricate nature of managing vast data, evolving technologies, and ensuring efficient data pipelines are some of the hurdles that make data engineering a demanding field. In this chapter, we will explain why data engineering is hard, and provide a foundation for the development of principles to be discussed in later chapters. We will also provide insight when navigating these challenges. Finally, we will provide an overview of the data engineering approaches you can use to scope out the current and near-future technology landscape.

We will discuss three of the main data engineering challenges:

• Platform architectures change rapidly
• There is a high cost and impact on a solution’s longevity from the strategy to buy rather than build
• The prolific evolving set of data repository...