Designing big data architectures

Big data solutions are comprised of data ingestion, storage transformation, data processing, and visualization in a repeated manner to run daily business operations. You can build these workflows using the open source or cloud technologies you learned about in previous sections.

First, you need to learn which architectural style is right for you by working backward from the business use case. You need to understand the end user of your big data architecture and create a user persona to understand the requirements better. To identify the key personas you are targeting with big data architecture, you need to understand some of the following points:

• Which teams, units, or departments inside your organization are they a part of?
• What is their level of data analysis and data engineering proficiency?
• What tools do they typically use?
• Do you need to cater to the organization’s employees, customers, or partners?
...