Exercise – build data pipeline orchestration using Cloud Composer

We will continue our bike-sharing scenario from Chapter 3, Building a Data Warehouse in BigQuery. Please finish Chapter 3 before going through this exercise.

This Cloud Composer exercise will be divided into five different DAG levels. Each DAG level will have specific learning objectives, as follows:

• Level 1: Learn how to create a DAG and submit it to Cloud Composer
• Level 2: Learn how to use operators
• Level 3: Learn how to use variables
• Level 4: Learn how to apply task idempotency
• Level 5: Handling DAG dependencies using an Airflow dataset

It’s important for you to understand that learning Airflow is as easy as Level 1 DAG. But as we go through each of the levels, you will see the challenges and opportunities we may have in practicing it.

In reality, you can choose to follow all of the best practices or none at all—Airflow won’t forbid you from doing that...