You're reading from Data Engineering with Google Cloud Platform A guide to leveling up as a data engineer by building a scalable data platform with Google Cloud

Product type Paperback

Published in Apr 2024

Publisher Packt

ISBN-13 9781835080115

Length 476 pages

Edition 2nd Edition

Languages

SQL

Tools

Google Cloud Platform

Concepts

Data Engineering

Author (1):

Adi Wijaya

View More author details

Table of Contents (19) Chapters

Preface

1. Part 1: Getting Started with Data Engineering with GCP FREE CHAPTER

2. Chapter 1: Fundamentals of Data Engineering

3. Chapter 2: Big Data Capabilities on GCP

4. Part 2: Build Solutions with GCP Components

5. Chapter 3: Building a Data Warehouse in BigQuery

6. Chapter 4: Building Workflows for Batch Data Loading Using Cloud Composer

7. Chapter 5: Building a Data Lake Using Dataproc

8. Chapter 6: Processing Streaming Data with Pub/Sub and Dataflow

9. Chapter 7: Visualizing Data to Make Data-Driven Decisions with Looker Studio

10. Chapter 8: Building Machine Learning Solutions on GCP

11. Part 3: Key Strategies for Architecting Top-Notch Solutions

12. Chapter 9: User and Project Management in GCP

13. Chapter 10: Data Governance in GCP

14. Chapter 11: Cost Strategy in GCP

15. Chapter 12: CI/CD on GCP for Data Engineers

16. Chapter 13: Boosting Your Confidence as a Data Engineer

17. Index

Why subscribe?

18. Other Books You May Enjoy

Cloud Composer 1 vs Cloud Composer 2

Currently, there are two main Cloud Composer versions, Cloud Composer 1 and Composer 2. Don’t be mistaken for the Airflow version, Airflow also has their versioning from the open-source. For example, currently, the latest Airflow version is 2.5.3.

What is the main difference between the two major Cloud Composer versions? There are some aspects that make Cloud Composer 1 and 2 different. But one of the most significant aspects is how the Airflow managed service works. In Cloud Composer 1, the number of Airflow workers is pre-defined for each environment. Cloud Composer 2 introduces the auto-scaling capability for the number of Airflow workers.

For example, in Cloud Composer 1, you need to choose the number of workers when you create the environment, let’s say three workers. This environment will always occupy three virtual machines for the environment, with or without workloads running on top of it. In other cases, if there are...