Working with data ingestion – an example pipeline

Let’s look at our data ingestion steps in practice. Assume that we do analytics for a factory specializing in Dutch delicacies: Stroopwafels. The CEO of this patisserie paradise has requested better insights into the effectiveness of providing Stroopwafel samples to potential customers. To answer their questions, we need to do the following:

1. Understand which potential customers (leads) have received samples. This data is available in a CRM tool where data from offline events and online requests is captured.
2. Understand whether these potential customers have purchased more than once. This data is only available in our highly secure, on-premise enterprise resource planning (ERP) tool.

We will go through the steps to get data from both systems.

Trigger

We have discussed with the CEO that daily updates are enough for the insights. We already have a scheduling tool such as Airflow available and will...