Chapter 11: CI/CD on Google Cloud Platform for Data Engineers

Continuous integration and continuous deployment (CI/CD) is a common concept for DevOps engineers, but most of the time, data engineers need to understand and be able to apply this practice to their development endeavors. This chapter will cover the necessary concepts of CI/CD and provide examples of how to apply CI/CD to our Cloud Composer example from Chapter 4, Building Orchestration for Batch Data Loading Using Cloud Composer. Upon completing this chapter, you will understand what CI/CD is, why and when it's needed in data engineering, and what GCP services are needed for it.

In this chapter, we will cover the following topics:

• Introduction to CI/CD
• Understanding CI/CD components with GCP services
• Exercise – Implementing CI using Cloud Build
• Exercise – Deploying Cloud Composer jobs using Cloud Build

Let's look at the technical requirements for the exercises in...