Exercise: Build data pipeline orchestration using Cloud Composer
We will continue our bike-sharing scenario from Chapter 3, Building a Data Warehouse in BigQuery.
This exercise will be divided into five different DAG levels. Each DAG level will have specific learning objectives, as follows:
- Level 1: Learn how to create a DAG and submit it to Cloud Composer.
- Level 2: Learn how to create a BigQuery DAG.
- Level 3: Learn how to use variables.
- Level 4: Learn how to apply task idempotency.
- Level 5: Learn how to handle late data.
It's important for you to understand that learning Airflow is as easy as learning Level 1 DAG, but as we go through each of the levels, you will see the challenges we may have in practicing it.
In reality, you can choose to follow all of the best practices or none of them—Airflow won't forbid you from doing that. Using this leveling approach, you can learn step by step from the simplest to the most complicated...
