You're reading from Architecting Google Cloud Solutions Learn to design robust and future-proof solutions with Google Cloud technologies

Product type Paperback

Published in Apr 2021

Publisher Packt

ISBN-13 9781800563308

Length 472 pages

Edition 1st Edition

Tools

Google App Engine

Concepts

Cloud Computing

Author (1):

Victor Dantas

View More author details

Table of Contents (17) Chapters

Preface

1. Section 1: Introduction to Google Cloud

2. Chapter 1: An Introduction to Google Cloud for Architects FREE CHAPTER

3. Chapter 2: Mastering the Basics of Google Cloud

4. Section 2: Designing Great Solutions in Google Cloud

5. Chapter 3: Designing the Network

6. Chapter 4: Architecting Compute Infrastructure

7. Chapter 5: Architecting Storage and Data Infrastructure

8. Chapter 6: Configuring Services for Observability

9. Chapter 7: Designing for Security and Compliance

10. Section 3: Designing for the Modern Enterprise

11. Chapter 8: Approaching Big Data and Data Pipelines

12. Chapter 9: Jumping on the DevOps Bandwagon with Site Reliability Engineering (SRE)

13. Chapter 10: Re-Architecting with Microservices

14. Chapter 11: Applying Machine Learning and Artificial Intelligence

15. Chapter 12: Achieving Operational Excellence

16. Other Books You May Enjoy

Designing and building data pipelines

A data pipeline for big data systems must be able to integrate, consolidate, and transform many different types of data from various sources. Other useful supporting capabilities include data discovery, preparation, and management. Let's look at each of these.

Data integration

Google Cloud has a suite of services for data integration functions. Cloud Dataflow is a unified stream and batch processing service based on Apache Beam. It is a fully managed, serverless service offering with horizontal autoscaling. It allows you to create Apache Beam pipelines. These are data integration pipelines that offer functionalities to read, transform, and ingest data. If you're unfamiliar with Apache Beam pipelines and their use cases, one simple example would be a pipeline that writes different data subsets to different data stores based on a filter. Suppose, for instance, that you have a database of books identified by their titles. Your pipeline...