The dynamic landscape of platform architectures poses both opportunities and challenges for data engineers. As systems and technologies rapidly evolve, there’s a pressing need for professionals in the field to stay abreast of these changes. Adapting to these shifts is not merely about understanding the newest tools or frameworks; it’s about foreseeing potential bottlenecks, ensuring system compatibility, and optimizing processes to accommodate these evolutions. Moreover, these frequent changes compel data engineers to embrace a continuous learning mindset, ensuring that their skills and knowledge remain relevant in an ever-shifting landscape.

In this section, you will learn about systemic changes in platform architectures in the last two decades, such as the move from SQL to NoSQL, the rise of big data, and the migration to the cloud. Please refer to Figure 2.2:

Figure 2.2 –...

Engineers are tasked with making it happen! But how they make it happen is subject to many constraints. The first is cost and the second is time. Issues such as the total cost of running and managing the solution over time and the feasibility of maintaining it operationally also come into focus. The TCO for a well-engineered data solution is affected by the extract, transform, and load (ETL)/extract, load, and transform (ELT) architecture and buy versus build tooling choices for selected adopted solutions and architecture patterns. Please refer to Figure 2.3:

Figure 2.3 – TCO mind map

After reading this section, you will have a greater understanding of ETL/ELT: what it is, its origins and historical evolution, why its costs are so high, and the advantages of build versus buy. You will also learn why legacy master data management architectures are no longer in vogue.

ETL architecture...

As data engineering terrain continually expands, a deluge of data repository patterns has flooded the market, each promising a seamless pathway to robust data architecture. However, amid this deluge, it’s imperative for data engineers, especially those embarking on this journey, to align their designs to the decided upon foundational principles. These principles will be applied to the solution as the big rocks to be handled first. These patterns must be effectively applied to the common challenges down to the smallest as the solution architecture is formulated. This section unravels these large items: data immutability, lineage tracking, data quality preservation, scalability, security and compliance, data discoverability and accessibility, and cost-efficiency (later chapters will depict the principles to be applied to them). Please refer to Figure 2.4 and Figure 2.5:

In this chapter, we explained the challenges data engineers will face when crafting a future-proof data engineered solution. Some core challenges have been outlined that will be faced when managing vast data, evolving technologies, and ensuring efficient data pipelines:

• Platform architectures change rapidly based on the cloud provider’s user demands, combined with shifting technology opportunities
• The total cost of your ownership (TCO) is high if you do not build a future-proof solution
• The data and system architecture that your design conforms to must be rational and handle important items first and not have them appear as obstacles later so that your future-proof goals are attainable

Data engineering remains a hard task, but you are going into the effort with your eyes open. A solid foundation has been laid for the principles to be discussed later. We provided an overview of data engineering approaches so that you can scope out the current and...