Chapter 5: Scalable Data Orchestration for Automation
Amazon Web Services (AWS) provides a rich set of native services to integrate a workflow. These workflows may involve multiple tasks that can be managed independently, thereby taking advantage of purpose-built services and decoupling them.
In this chapter, we will primarily focus on workflows such as extract, transform, load (ETL) processes that are used to refresh a data warehouse. We will illustrate different options that are available using the individual recipes, but these are interchangeable depending on your use case. For example, in your workflow, you can call an AWS Python shell (https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html) instead of the Amazon Redshift Data application programming interface (API) in cases where you might want to reuse your existing Python code base.
The following recipes are discussed in this chapter:
- Scheduling queries using the Amazon Redshift query editor
- Event...