Summary

In this chapter, we explained the challenges data engineers will face when crafting a future-proof data engineered solution. Some core challenges have been outlined that will be faced when managing vast data, evolving technologies, and ensuring efficient data pipelines:

• Platform architectures change rapidly based on the cloud provider’s user demands, combined with shifting technology opportunities
• The total cost of your ownership (TCO) is high if you do not build a future-proof solution
• The data and system architecture that your design conforms to must be rational and handle important items first and not have them appear as obstacles later so that your future-proof goals are attainable

Data engineering remains a hard task, but you are going into the effort with your eyes open. A solid foundation has been laid for the principles to be discussed later. We provided an overview of data engineering approaches so that you can scope out the current and...