Summary
Throughout this chapter, we have explored the capabilities of CI/CD tools in automating code deployment, how CI/CD tools can optimize our development workflows, and how they can be specifically tailored to various ETL processes. We also walked through a practical implementation of various AWS CI/CD services – CodePipeline, CodeDeploy, and CodeCommit – to create a foundation for your production-grade environment so that you can create more efficient, reliable, and scalable deployment workflows.
In the next chapter, we will transition from learning how to implement CI/CD pipelines to creating an effective orchestration of ETL pipeline deployment environments. Using CI/CD pipelines as the “silent structure,” we will learn how to refactor a basic data pipeline so that it’s both resilient and scalable by leveraging scaling and orchestration strategies.
