Introduction

In previous chapters, we introduced different databases for different business-use cases. We also introduced the next-generation compute engine Spark for big data analytics. With these tools, we now have all the necessary building blocks for composing any AI data pipeline:

Figure 9.1: A representative flow chart for a typical data pipeline

A typical data pipeline (not limited to AI) looks like the following:

1. Collect user feedback from a user application.
2. Store all user feedback and data in a data storage system.
3. Extract raw user data from the data storage system.
4. Preprocess raw data into a predefined format so that data science/AI applications can process it.
5. Cook the processed data into a higher-level view so that business people such as product managers can digest it and make data-informed decisions.

Let's imagine you are working in a data-driven company such as Netflix. Data scientists are building data...