Examining examples of real-world data pipelines
The data pipeline examples that we have used in this book have been based on common types of transformations and pipelines, but they have been relatively simple examples. As you can imagine, in large organizations, the types of data pipelines that are built can be a lot more complex and may end up processing extremely large sets of data.
In this section, we will examine two examples of more complex data engineering pipelines from two very well-known organizations – Spotify and Netflix. Both of these companies have public blogs that cover software and data engineering, and the details provided about their pipelines in this section have been taken from the public information that's been made available in a variety of blog posts and articles.
A decade of data wrapped up for Spotify users
Every year, for the past few years, the music streaming service Spotify has used the extensive data they have on their user's...
