Summary
This was an extremely important chapter for several reasons. Using a few examples, we learned about the common challenges that are faced in the world of data engineering. Dealing with challenges such as schema evolution and governance are critical in modern data engineering projects. These challenges have evolved over time, and so have the techniques and tools that make the job of the data engineer a little easier.
In the next chapter, we will look at DevOps essentials for data engineers. DevOps is quickly becoming an essential add-on skill for data engineers, so we will learn how to automatically provision cloud resources using the Infrastructure as Code (IaC) paradigm. Using the power of IaC, data engineers can spin up data pipelines that use cloud resources in a fast, repeatable, and secure fashion.
