Chapter 12: Continuous Integration and Deployment (CI/CD) of Data Pipelines
Our data journey is finally approaching its destination. As the new era of analytics takes over, the demand for data engineers will continue to grow, and so will the amount of code that they will produce. The ever-increasing demand for developing, managing, and deploying large code sets is already testing the limits of modern data engineers.
Luckily, a modern trend is fast emerging that has the potential of taking a lot of burden off poor data engineers. In this chapter, we will learn about code delivery automation using CI/CD pipelines. In short, CI/CD is a collection of practices that's used to integrate and deliver code faster using small atomic changes.
In this chapter, we will cover the following topics:
- Understanding CI/CD
- Designing CI/CD pipelines
- Developing CI/CD pipelines
