Section 3: Data Engineering Challenges and Effective Deployment Strategies

This part of the book educates you on the challenges faced by data engineering professionals. The concepts of Delta Lake are introduced. Lastly, a number of data engineering challenges will be highlighted with real-life examples of how Delta Lake can help deal with these issues effectively.

This section also educates you on effective deployment strategies for a data lake. Once the individual code components dealing with the various stages of data engineering have been built, these strategies play an important role in provisioning the cloud resources and deploying the data pipelines in a repeatable and continuous fashion.

This section contains the following chapters:

• Chapter 9, Deploying and Monitoring Pipelines in Production
• Chapter 10, Solving Data Engineering Challenges
• Chapter 11, Infrastructure Provisioning
• Chapter 12, Continuous Integration and Deployment (CI/CD) of Data Pipelines...