Exam scenario
MegaCorp Inc. is building a new data analytics capability to help understand its marketing campaigns' effectiveness and how they relate to product sales.
Marketing campaign data is exported daily and stored as flat CSV files. Sales data is exported overnight from the sales database into a normalized data warehouse database.
The management team would like data to be automatically imported and aggregated, and then modeled. It is expected that large amounts of data will be processed, and this needs to be performed relatively quickly. The data analytics teams are seasoned developers who are currently using the latest version of Spark.
Design an end-to-end solution that can accommodate the management team's requirements.