Chapter 8: Approaching Big Data and Data Pipelines

Big data plays an essential role in modern enterprise architectures and practices. While traditionally, businesses have been able to store and use their data entirely using relational database management systems, datasets have become too diverse and voluminous to be managed with traditional methods. For data-driven companies, being able to unlock value from data is almost a matter of survival.

In this chapter, you will learn about core big data concepts and services and how to design solutions using data pipelines for ingestion, streaming, batch processing and analytics. You will also learn about common architectural patterns and get hands-on experience with building a big data solution on GCP. By the end of this chapter, you should have acquired a good understanding of the big data ecosystem on GCP and feel confident with designing solutions that solve real challenges faced by data-driven businesses.

In this chapter, we&apos...