Chapter 4: Building Orchestration for Batch Data Loading Using Cloud Composer
The definition of orchestration is a set of configurations to automate tasks, jobs, and their dependencies. If we are talking about database orchestration, we talk about how to automate the table creation process.
The main objects in any database system are tables, and one of the main differences between an application database and a data warehouse is the creation of tables. Compared to tables in application databases, where tables are mostly static and created to support applications, tables in data warehouses are dynamic. Tables are products, collections of business logic, and data flows.
In this chapter, we will learn how to orchestrate our data warehouse tables from Chapter 3, Building a Data Warehouse in BigQuery. We will learn how to automate the table creations using a Google Cloud Platform (GCP) service called Cloud Composer. This will include how to create a new Cloud Composer environment...
