Developing the architecture and choosing the right tools  

In data engineering, after the data has been successfully ingested and stored in a data lake or a data warehouse, often, it needs to be mined and stored for specific needs in a more sorted and customized form for reporting and analysis. In this chapter, we will discuss such a problem where a huge volume of data needs to be analyzed and stored in a more customized format for a specific downstream audience.

Problem statement

Let’s assume that an e-commerce firm, ABC, wants to analyze various kinds of user interaction on its products and determine the top-selling products for each category each month. They want to provide incentives to the top-selling products in each category. They also want to provide special offers and marketing promotion tools to products with top view-to-sale ratios but are not the top-selling products. In addition, they want to market seller tools and training, as well as marketing...